Building AI Agents with Open Source Models: From RAG Pipelines to Autonomous Workflows

AI agents are moving from demo to production, and open source models have caught up enough to power most of the agentic workflows mid market brands actually need. The architecture patterns are different from simple prompt and response. Here's how to build them right.

The core shift: instead of calling a model once and using its output directly, agentic systems call models repeatedly in a loop, reasoning about what to do next, executing actions via tool calls, evaluating results, and iterating until the task is complete. This loop architecture changes what matters in model selection. Raw benchmark scores matter less. Reliability, tool use accuracy, and cost per step matter more.

Why Open Source Models Work for Agents

The conventional wisdom says agents need the most powerful model available because agentic loops amplify errors. A bad decision in step 2 compounds through steps 3 through 10. This is true in theory. In practice, it depends entirely on the complexity of the decision space.

For most ecommerce agentic workflows, the decision space is constrained. An inventory management agent is choosing between reorder, hold, markdown, and escalate. A customer service agent is routing between known resolution paths. A content generation agent is following a defined template. These are not open ended reasoning problems. They're structured workflows where a fine tuned 8B model makes the right decision 97% of the time, and the 3% failure rate is handled by fallback logic, not a more expensive model.

The math confirms this. An agentic workflow that takes 8 steps per task, running 1,000 tasks per day, generates 8,000 model calls daily. At proprietary API pricing, that's $400 to $800 per day depending on token volume. The same workload on a self hosted Llama 4 Scout model costs $25 to $40 per day in compute. Over a year, the difference is $130,000 to $275,000. That funds an entire engineering position.

Architecture Pattern 1: RAG Powered Product Experts

Retrieval Augmented Generation is the foundation for any agent that needs to answer questions about your specific products, policies, or operations. The pattern is well established, but the implementation details matter enormously for production quality.

The Stack

Embedding model: Use an open source embedding model, not a proprietary one. Models like bge-large-en-v1.5 or GTE-Qwen2 produce embeddings that match or exceed OpenAI's text-embedding-3-large on retrieval benchmarks. They run on CPU, cost nothing per call when self hosted, and eliminate a dependency on external APIs.

Vector store: For mid market scale (under 5 million documents), pgvector running inside your existing PostgreSQL instance is the right choice. You don't need a dedicated vector database. Adding a vector column to your existing product table keeps your data model simple and your operational surface small. For larger scale or more sophisticated retrieval patterns, Qdrant or Weaviate are the purpose built options.

Generation model: Llama 4 Scout or Mistral Small for the generation step. The context window needs to hold 3 to 5 retrieved documents plus the conversation history. Both models handle 32K tokens comfortably, which is more than enough for this pattern.

The Implementation That Actually Works

Most RAG tutorials show a simple retrieve then generate pipeline. Production RAG for ecommerce needs three additions:

Hybrid retrieval. Pure vector similarity search misses exact matches. When a customer asks for SKU "BLK-WIDGET-42," semantic search might return similar products instead of the exact one. Combine vector search with keyword search (BM25) and let a simple reciprocal rank fusion merge the results. PostgreSQL full text search plus pgvector gives you both in one query.

Metadata filtering. Before running similarity search, filter by structured attributes. If the customer is asking about return policies for electronics, filter your document index to policy documents tagged with the electronics category. This eliminates irrelevant results that happen to be semantically similar.

Answer grounding verification. After the model generates a response, run a lightweight verification step: does the response contain claims that are actually supported by the retrieved documents? A small classifier model (even a fine tuned BERT) can flag hallucinated product specs or made up policy details before the answer reaches the customer. This is the step most teams skip, and it's the one that prevents the most costly errors.

Architecture Pattern 2: Tool Using Agents for Operations

The next level beyond RAG is agents that don't just answer questions but take actions. Open source models with tool use capabilities can call APIs, update databases, trigger workflows, and make decisions based on real time data.

Defining the Tool Interface

Open source models have gotten significantly better at structured tool use. Llama 4's instruction tuned variants and Mistral's function calling models both support a tool use format where you define available functions as JSON schemas, and the model outputs structured function calls.

For an inventory management agent, the tool definitions might include:

• check_inventory(sku, location) returns current stock levels
• create_purchase_order(sku, quantity, supplier) initiates a reorder
• update_price(sku, new_price, reason) adjusts pricing
• send_alert(channel, message, severity) notifies the operations team
• query_sales_history(sku, period) pulls historical sales data

The agent receives a trigger (low stock alert, scheduled audit, anomaly detection), reasons about the situation using its tool outputs, and executes the appropriate response.

The Control Loop

The critical architectural decision is how much autonomy the agent gets. We use a three tier model:

Tier 1: Fully autonomous. The agent acts without human approval. Reserved for low risk, reversible actions: sending internal alerts, generating reports, updating non customer facing metadata. The model needs to be right 99%+ of the time for these actions.

Tier 2: Propose and confirm. The agent makes a recommendation and queues it for human approval. Used for medium risk actions: price changes, purchase orders under $5,000, customer communication drafts. The human reviews a structured summary, not the raw model output.

Tier 3: Escalate. The agent recognizes it's outside its competence and routes to a human with full context. Used for novel situations, high value decisions, or when the model's confidence score falls below a threshold.

This tiered approach lets you deploy agents quickly on Tier 2 and 3 actions, then gradually promote actions to Tier 1 as you build confidence in the model's reliability for specific decision types.

Why Open Source Matters Here

Tool use agents make many sequential model calls per task. Each call is relatively simple ("given these inventory levels and this sales trend, should I reorder?") but the volume is high. Open source models let you run this loop at $0.001 to $0.003 per call instead of $0.01 to $0.03 per call. That 10x cost reduction is the difference between an agent that's economically viable at scale and one that's a money pit.

More importantly, self hosted models give you deterministic availability. An agentic loop that's halfway through a 12 step workflow can't tolerate a rate limit error or API timeout. When the model is running on your infrastructure, you control the availability SLA.

Architecture Pattern 3: Multi Agent Orchestration

The most powerful pattern, and the one most teams should wait to implement, is multiple specialized agents collaborating on complex tasks.

How It Works

Instead of one large model handling everything, you build a system of small, specialized agents:

• Router agent (tiny model, 1 to 3B parameters) classifies incoming requests and routes to the appropriate specialist
• Product expert agent (RAG powered, 8B model) answers product questions using your catalog and knowledge base
• Order management agent (tool using, 8B model) handles order status, modifications, returns using your OMS API
• Escalation agent (larger model, 30B+) handles complex situations that other agents can't resolve

Each agent is fine tuned for its specific domain. The router agent's only job is classification; it's fast and cheap. The specialist agents go deep on their domain. The escalation agent is the only one that needs the expensive, high capability model.

The Orchestration Layer

Build the orchestration layer as explicit code, not as a model prompt. Use a framework like LangGraph, CrewAI, or plain Python with a state machine. The orchestration logic should be:

• Deterministic. Routing rules are code, not model decisions (except for the router agent's classification)
• Observable. Every agent call, tool use, and decision is logged with full context
• Interruptible. Any workflow can be paused, inspected, and resumed by a human
• Timeout bounded. No agentic loop runs indefinitely; set hard limits on steps and wall clock time

The biggest failure mode in multi agent systems is recursive delegation. Agent A calls Agent B, which calls Agent A, which calls Agent B. Prevent this architecturally by enforcing a directed acyclic graph of agent dependencies.

When to Use Multi Agent vs. Single Agent

Use a single agent when:

• The task domain is narrow (just inventory, just customer service)
• You have fewer than 10 tools the agent needs access to
• Request volume is under 500 per day

Use multi agent when:

• The task crosses multiple domains (a customer question that involves product knowledge, order status, and return policy)
• You have 20+ tools and a single model can't reliably select the right one
• You need different quality/cost trade offs for different sub tasks

Most mid market brands should start with single agent patterns and only move to multi agent when they hit specific scaling or accuracy limitations.

The Evaluation Problem

The hardest part of building AI agents is knowing whether they're working correctly. Unlike a classification model where accuracy is a single number, agent quality is multi dimensional: Did it take the right actions? In the right order? With the right parameters? Did it know when to stop?

Build evaluation datasets from production traffic. Log every agent run with inputs, intermediate steps, and outputs. Have domain experts review a random sample weekly and label each run as correct, partially correct, or incorrect. Track these metrics over time. When you fine tune or update the model, re run the evaluation dataset and compare.

Automated evaluations help for regression testing, but human review is irreplaceable for catching subtle quality degradation.

How Contra Collective Builds This

We architect and implement agentic AI systems for mid market ecommerce brands, using open source models where they make sense and proprietary models where they don't. Our approach starts with the business workflow, not the technology. We map the decision points, define the autonomy tiers, and build the agent architecture around your actual operational needs. Reach out if you're evaluating agentic AI for your operations.

Final Thoughts

Open source models have made AI agents economically viable for mid market brands. The architecture patterns (RAG pipelines, tool using agents, multi agent orchestration) are proven and well documented. The challenge isn't technical capability; it's operational discipline. Build agents with explicit control boundaries, invest in evaluation infrastructure, and start with constrained, well defined workflows before attempting open ended autonomy. The brands that get agentic AI right will have a structural operational advantage. The ones that skip the engineering fundamentals will have expensive, unpredictable systems that erode trust in AI across their organization.