AI Agents · 11 min read
The AI Agent Stack: What Tools Actually Work Together

The agent stack is the set of tools, models, memory systems, and integration layers that make an AI agent functional in production. Most operations leads pick tools in isolation and wonder why nothing coheres. The answer is almost always architectural: the layers were never designed to talk to each other, and no one mapped the data flows before the first line of code was written.
Why the Agent Stack Is Not Just a Model Choice
The model is the least interesting decision in a well-designed agent stack. GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro — they are all capable enough for most business tasks. What separates a working agent from a demo is everything around the model: how it receives context, what tools it can call, how it stores and retrieves information across sessions, and how it hands off to other systems or humans when it hits a limit.
Operations leads who treat the agent stack as a model-selection exercise end up with expensive chatbots. The real question is: what does the agent need to do, and what infrastructure does that require?
The Five Layers of a Working Agent Stack
A production-grade agent stack has five distinct layers. Each one can fail independently, and a failure in any layer degrades the whole system. Understanding the layers is the first step to evaluating whether a vendor, a platform, or an in-house build will hold up under real operational load.
- Model layer: The LLM that reasons, generates, and decides. This is the brain, not the whole body.
- Tool layer: The functions and APIs the model can call — CRM reads, calendar writes, database queries, web searches.
- Memory layer: Short-term context (the conversation window) and long-term storage (vector databases, structured logs, user profiles).
- Orchestration layer: The logic that routes tasks, manages multi-step workflows, and decides when to escalate to a human or a different agent.
- Integration layer: The connectors between the agent and your existing systems — your CRM, your helpdesk, your data warehouse, your communication channels.
The Model Layer: What Actually Matters
For most business use cases, the model decision comes down to three variables: context window size, tool-calling reliability, and cost per token at your expected volume. A 200,000-token context window matters if your agent needs to reason over long documents or multi-session histories. Tool-calling reliability matters more than raw benchmark scores — a model that hallucinates function arguments breaks workflows in ways that are hard to catch and expensive to fix.
Context Window vs. Retrieval
A large context window is not a substitute for a proper retrieval system. Stuffing 50,000 tokens of CRM history into every prompt is slow and expensive. The right pattern is retrieval-augmented generation (RAG): store long-term data in a vector database, retrieve only the relevant chunks at inference time, and keep the context window lean. This is where the memory layer and the model layer have to be designed together, not bolted together after the fact.
Model Cost at Scale
At low volume, model cost is irrelevant. At 50,000 agent calls per month — a realistic number for a mid-market operations team — the difference between a $0.003 and a $0.015 per-1K-token model is material. Run the math before you commit to a model. The Anthropic model overview is a useful reference for understanding the capability-cost tradeoffs across the Claude family, which is one of the more reliable options for tool-heavy agent work.
Tool Use and Function Calling
Tool use is what separates an agent from a chatbot. An agent that can only generate text is a sophisticated autocomplete. An agent that can read your CRM, write to your calendar, query your database, and send a Slack message is an operator. The agent stack’s tool layer defines the scope of what the agent can actually affect in the world.
Function calling — the mechanism by which a model decides to invoke a tool, constructs the arguments, and processes the result — is the most failure-prone part of most agent stacks. The model has to understand what each tool does, when to use it, and how to handle errors when the tool returns an unexpected result. Poorly documented tool schemas are the single most common cause of agent failures in production. Write your tool descriptions as if you are explaining them to a careful but literal-minded contractor, because that is exactly what you are doing.
Tool Schema Design
Every tool in the agent stack needs a clear name, a precise description, and typed parameters with explicit constraints. Vague schemas produce vague calls. If your update_crm_record function accepts a free-text status field, the model will invent values. Enumerate the valid options. Validate inputs before execution. Log every tool call and its result. This is not optional — it is the only way to debug an agent that is misbehaving in production.
Memory and Context Management
The agent stack’s memory layer is what makes an agent feel like a colleague rather than a stranger at every interaction. Without persistent memory, every session starts from zero. The agent cannot remember that a customer already complained about the same issue last week, or that a lead said they were evaluating three vendors. That context is exactly what makes a human operator valuable — and it is exactly what most agent deployments throw away.
There are three types of memory worth designing for. Episodic memory stores what happened in past sessions. Semantic memory stores facts about entities — customers, products, accounts. Procedural memory stores how to do things — the agent’s learned preferences and workflow patterns. Most agent stacks implement only episodic memory, and only imperfectly. If you want to understand why memory architecture is the feature that changes the economics of agent deployment, the analysis at Why AI Agent Memory Is the Feature That Changes Everything is worth reading before you finalize your stack design.
Orchestration and Routing
Single-agent systems hit a ceiling fast. A single agent trying to handle lead qualification, contract drafting, and customer support simultaneously will either be too generic to be useful or too complex to be reliable. The answer is orchestration: a routing layer that receives an incoming task, classifies it, and dispatches it to the right specialist agent.
Orchestration frameworks like LangGraph, CrewAI, and custom-built routers each make different tradeoffs. LangGraph gives you fine-grained control over state and transitions — useful when your workflows are complex and deterministic. CrewAI is faster to prototype but harder to debug at scale. Custom routers are the most reliable in production but require the most engineering investment. The decision depends on how much workflow complexity you are managing and how much engineering capacity you have to maintain the system. For a deeper look at how this decision compounds over time, see Why Agent Orchestration Is the New Tech Stack Decision.
Integration Layer: Where Most Stacks Break
The integration layer is where the agent stack meets your existing business systems. This is where most deployments fail — not because the model is wrong or the orchestration is broken, but because the connectors are fragile, the authentication is misconfigured, or the data schemas between systems do not match.
The Model Context Protocol (MCP) is an emerging standard for agent-to-system integrations that is worth understanding. It defines a common interface for how agents discover and call tools across different systems, which reduces the custom glue code required for each new integration. Whether you adopt MCP or build custom connectors, the principle is the same: treat every integration as a contract with explicit inputs, outputs, and error states. An agent that cannot gracefully handle a failed API call will fail loudly in front of a customer.
- CRM integrations: Read and write access, not just read. An agent that can only look up records but cannot update them is half an operator.
- Communication channels: Email, Slack, SMS, and voice each have different latency and formatting requirements. Design for the channel, not just the content.
- Data warehouses: Agents that need to reason over historical data need structured query access, not raw table dumps. Build a query interface, not a data pipe.
- Human escalation paths: Every agent stack needs a defined handoff protocol. When the agent cannot resolve something, where does it go, and what context does it pass along?
Agent Stack Comparison: Assembled vs. Purpose-Built
| Dimension | Assembled Stack (off-the-shelf tools) | Purpose-Built Stack (custom architecture) |
|---|---|---|
| Time to first deploy | 2–6 weeks | 8–16 weeks |
| Integration flexibility | Limited to supported connectors | Any system with an API |
| Maintenance overhead | Vendor-managed, but version lock risk | Team-managed, full control |
| Cost at scale | Predictable SaaS pricing, often high per-seat | Infrastructure cost only, lower at volume |
| Failure visibility | Black-box logs, limited debugging | Full observability if instrumented correctly |
| Workflow complexity ceiling | Low-to-medium | High |
What an Operations Lead Should Actually Evaluate
Most agent stack evaluations focus on the demo. The demo always works. What you need to evaluate is the failure surface: what happens when the model returns a malformed tool call, when an API times out, when a user provides ambiguous input, or when the agent hits a task it was not designed for. A robust agent stack handles these cases gracefully. A fragile one fails silently or loudly, and you find out from a customer.
Ask every vendor or internal team three questions. First: how does the agent handle a tool call that returns an error? Second: what is the escalation path when the agent’s confidence is low? Third: how do you observe and debug agent behavior in production? If the answers are vague, the agent stack is not production-ready. For a grounded view of what a well-designed agent actually delivers operationally, What a Revenue-Generating AI Agent Actually Does is a useful reference point.
The agent stack is not a one-time decision. It is an architecture that you will maintain, extend, and debug for as long as the agent is running. Build it like infrastructure, not like a prototype.
If you are mapping your agent stack and want a second opinion on the architecture before you commit, Studio Máté is worth a conversation.
FAQ
What is an agent stack?
An agent stack is the full set of components that make an AI agent functional in production: the language model, the tools it can call, the memory systems that store context, the orchestration logic that routes tasks, and the integration layer that connects the agent to your existing business systems. Each layer can fail independently, so all five need to be designed together.
How do I choose the right model for my agent stack?
Evaluate models on context window size, tool-calling reliability, and cost per token at your expected monthly volume — not on general benchmark scores. For most business operations use cases, reliability of function calling matters more than raw capability. Run a structured test with your actual tool schemas before committing to a model.
Why do most agent stacks fail in production?
The most common failure points are poorly designed tool schemas, fragile integrations with existing systems, and the absence of a human escalation path. Agents that work in demos fail in production because the demo does not surface edge cases: ambiguous inputs, API timeouts, or tasks outside the agent’s designed scope. Build for failure modes, not just the happy path.
What is the difference between an assembled and a purpose-built agent stack?
An assembled stack uses off-the-shelf platforms and connectors — faster to deploy but limited in flexibility and often expensive at scale. A purpose-built stack is custom-engineered for your specific workflows — slower to build but more reliable, more observable, and cheaper per call at volume. The right choice depends on your workflow complexity and your team’s engineering capacity.
How does memory fit into the agent stack?
Memory is what makes an agent useful across sessions rather than just within a single conversation. A well-designed agent stack separates short-term context (the active conversation window) from long-term storage (a vector database or structured log of past interactions and entity facts). Without persistent memory, every agent session starts from zero, which eliminates most of the operational value the agent could deliver.