TL;DR
AI agent architecture is everything you build around a language model so it can do real work. There are seven parts to it. The model does the thinking. A planner decides the next step and when to stop. Memory keeps track of the task. Retrieval fetches facts the model never learned. Tools are the only things the agent can touch. Guardrails cap what those tools may do. A trace records every step for later. The model is the easy part. Agent projects usually stall on the other six.
Key Takeaways An AI agent architecture has seven layers. Model, planning, memory, tools, retrieval, guardrails, and observability. What an agent can physically do is set by the tool layer, which makes the tool registry the real security boundary. Anthropic and Microsoft publish five architecture patterns each, under different names, describing largely the same five shapes. Most enterprise workloads want a deterministic workflow with agent reasoning only at the genuinely ambiguous steps. The classical agent structures from AI textbooks still map cleanly onto modern layers, which is useful when you are choosing how much autonomy to grant. Observability and an audit trail belong in the architecture from day one, because you cannot retrofit a record of decisions that were never logged. Watch on YouTube
How Enterprise AI Agents Are Designed for Real Business Decisions
Kanerika’s team walks through the design choices that separate an agent demo from an agent a business will actually let near a production system, and where the architecture has to carry the weight.
The Demo Always Works. Production Is Where the Architecture Shows. A procurement agent aces its demo. It reads a contract, pulls the vendor record, checks the payment terms, and drafts a summary in eleven seconds. Three weeks later the same agent approves a renewal at last year’s pricing, because the record it read was cached, nobody had told it that stale context was disqualifying, and no log captured which document it actually used.
Nothing about that failure was a model problem. A larger model would have produced the same confident answer from the same wrong input. The failure lives in the layers around the model, which is exactly what architecture means here.
That gap between a working demo and a system a business will trust is the whole subject of this article.
What Is AI Agent Architecture? AI agent architecture is the structural design of the components that surround a language model and let it perceive a situation, decide on a course of action, take that action against real systems, and observe the result. The model supplies reasoning. The architecture, meanwhile, supplies everything that makes the reasoning safe, repeatable, and connected to something real.
The distinction that matters most comes from Anthropic’s engineering team, which draws the line at who controls the path. Anthropic defines it plainly . “Workflows are systems where LLMs and tools are orchestrated through predefined code paths. Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks.”
That single sentence is the fork in the road for every architecture decision that follows. A workflow is predictable and testable because a human wrote the sequence. An agent, by contrast, is adaptable and harder to test, since the sequence is chosen at runtime.
The Difference Between a Model, an Assistant, and an Agent Three things get called AI in the same conversation, and they have three different architectures.
A model takes text in and returns text out. That is, it has no memory between calls, no ability to reach any system, and no way to verify anything it says. An assistant wraps a model in a conversation loop and usually a retrieval step, so it can answer from your documents and remember the current thread. Even so, it still only produces words.
An agent adds the ability to act. It can call an API, write to a database, file a ticket, or send a message, and moreover it decides for itself which of those to do. Everything hard about agent architecture follows from that one addition, because a system that can act can also act wrongly. The same line separates an agent from a scripted bot, which is covered in our comparison of AI agents and chatbots . If the boundary between these three is still fuzzy, our breakdown of AI agents versus AI assistants covers it in more detail.
AI Agent Architecture Diagram: The Full Stack at a Glance The clearest way to hold an agent architecture in your head is as a stack, read from the bottom up. Each layer depends on the one below it and can fail independently of the rest.
The seven layers of an AI agent architecture, read from the bottom up. Read from the bottom, the stack answers four questions in order. First, what can this agent reach. Second, what does it know. Third, how does it decide. Finally, who is watching. An architecture review that cannot answer all four is not finished.
Why Architecture Decides Whether an Agent Survives Production Pilot agents are graded on whether the output looks right. Production agents, by contrast, are graded on what happens the one time in two hundred when the output is wrong.
Those are, in practice, different engineering problems. The first rewards a capable model and a good prompt. The second rewards bounded permissions, a retrieval layer that returns fresh data, a stopping condition, and a trace you can read afterwards. Teams that treat the second set as a later hardening phase usually find that the record they need to debug the incident was never written. Our account of the evolution of AI agents traces how the bar moved from answering to acting.
The Seven Layers of an AI Agent Architecture Ranking pages describe anywhere from four to seven components, and the disagreement is mostly about where to draw boundaries rather than what exists. Seven layers is nevertheless the division that holds up in production, because each one owns a distinct failure mode and a distinct owner inside an enterprise.
Layer 1: The Model Layer The model layer is the reasoning engine. In practice it is rarely one model. In practice, production systems route cheap, high-volume steps such as classification and extraction to a small model, while reserving a frontier model for planning and synthesis.
The architectural decisions here are latency, cost per task, and where inference physically runs, since that matters whenever the input contains regulated data. Model choice gets the most attention in vendor conversations. It is nevertheless the easiest thing to change later, because a well-designed agent treats the model as a replaceable component behind an interface.
Layer 2: The Planning and Reasoning Layer The planning layer converts a goal into a sequence of steps. Specifically, it decomposes the task, picks the next action, and decides when the work is done.
This layer holds the stopping condition, which is the single most commonly missing piece of an agent architecture. Without an explicit budget on steps, tokens, or wall-clock time, an agent that cannot solve a task will keep trying variations until something external kills it. That is how a five-cent task becomes a forty-dollar one.
Give that budget real numbers, and enforce them in the runtime where no prompt can talk its way past them. For a bounded enterprise task, a ceiling of 10 to 15 tool calls per run, 60 to 90 seconds of wall-clock time, and a token cap at roughly three times the measured median run is a defensible starting point. Log every run that hits a ceiling as a failure to review. A rising rate of budget exhaustion is the earliest signal that retrieval quality or tool design has drifted.
Layer 3: The Memory Layer, Short-Term State and Long-Term Recall Memory splits into two structures that behave nothing alike, and conflating them causes real damage.
Short-term memory holds the current task. The conversation so far, intermediate results, which tools have already been called. It lives for the duration of one session, and therefore should be discarded when that session ends.
Long-term memory persists across sessions. User preferences, prior decisions, learned corrections. It is a database, and it inherits every obligation a database carries. Retention limits, deletion rights, access control, and a defensible answer to why a piece of personal data is still there. Google Cloud’s architecture guidance treats the two as separate components for exactly this reason.
The common design error is letting long-term memory grow without a policy, on the assumption that more history means better answers. Past a point it means slower retrieval, higher cost per call, and a compliance surface nobody scoped.
Layer 4: The Tool and Action Layer The tool layer is where an agent stops producing text and starts changing things. It holds the API clients, database connections, and application integrations the agent can invoke.
One principle governs the whole layer. The model proposes an action, the tool layer decides whether that action is possible at all. For instance, a model cannot delete a record if no deletion tool is registered, no matter how convincingly it argues for one. That makes the tool registry a security boundary. Permissions belong there, where no phrasing in a prompt can override them.
Tool interfaces have been standardising around the Model Context Protocol , an open protocol for how applications supply context and tools to language models. Adopting it means a tool you expose once works with any MCP-aware agent, instead of being rebuilt per framework. Our walkthrough of MCP and context-aware agents goes deeper on the mechanics.
Google Cloud’s architecture guidance also names a failure mode worth designing against from the start, which it calls tool bloat. Too many tool definitions degrade selection accuracy, because the model is choosing from a menu it cannot hold in working memory. The same guidance recommends keeping each tool to fewer than five parameters. The remedy sits in the architecture. Group tools behind a smaller set of task-scoped interfaces. Kanerika’s pillar guide to agentic AI covers where this sits in the wider stack.
On-Demand Webinar
Model Context Protocol: The Key to Building Context-Aware AI Agents
A working session on how MCP standardises the tool layer, what it changes about connecting agents to enterprise systems, and where teams get the permission model wrong.
Watch On Demand →
Layer 5: The Knowledge and Retrieval Layer Retrieval gives the agent facts it was not trained on. These include vector indexes, keyword search, structured queries against a warehouse, and increasingly a knowledge graph that encodes how entities relate.
Agent retrieval differs from ordinary search in one way that changes the design. That is, the agent decides what to look for mid-task, based on what it has already found. So the retrieval layer needs to answer questions the system designer never anticipated, which puts the weight on index coverage and freshness rather than on query tuning. Deciding what belongs in the window at each step is its own discipline, covered in agentic context engineering . If retrieval is the part of your stack under pressure, agentic RAG covers the patterns that hold up.
Layer 6: The Guardrail and Policy Layer Guardrails are the constraints an agent cannot reason its way around, and they only work while they sit outside the model instead of inside the prompt.
Three kinds matter. First, input constraints reject work the agent should not attempt. Second, action constraints cap what a single tool call can do, such as a spend ceiling or a row limit. Third, output constraints check the result before it reaches a person or another system. Human approval checkpoints are the strongest version of an action constraint. Indeed, they are the reason most regulated deployments ship at all.
OWASP now names the underlying risk directly. Excessive Agency sits in the Top 10 for LLM applications and breaks into excessive functionality, excessive permissions, and excessive autonomy. Since those three map onto the tool registry, the credential model, and the approval policy, the list works as a review checklist instead of an abstract warning.
Layer 7: The Observability and Evaluation Layer An agent decides at runtime, so a record of the decision is the only way to explain the outcome afterwards. At minimum, the trace needs the goal, the plan, every tool call with its arguments and response, every retrieval hit, the token and cost figures, and the final action.
A Minimum Trace Schema A workable trace schema is short, and it is worth writing down before the first agent ships. One record per turn, carrying run_id, turn_index, user_id, agent_id, goal_text, plan_step, tool_name, tool_args_hash, tool_result_status, retrieved_doc_ids, prompt_tokens, completion_tokens, latency_ms, cost_usd, guardrail_verdict and approved_by. Hash the arguments rather than storing them raw wherever they can contain personal data, and keep the document ids so a reviewer can reconstruct what the agent actually read.
Scoring Behaviour, Not Output This layer is also where evaluation lives. Traditional software testing asserts on outputs for known inputs. Agent evaluation has to score behaviour, which means a held-out task set, a rubric, and a scheduled run, treated as part of the system rather than as QA. A set of 50 to 200 recorded real tasks, replayed nightly and scored on task completion and tool-choice correctness, catches drift on most enterprise workloads. Our piece on AI agent observability goes into the instrumentation.
Observability cannot be retrofitted, because the incident you need to explain was either logged or it was not.
Table 1. The seven layers, what each one owns, and what breaks when it is missing.
Layer What it owns What breaks without it Who usually owns it Model Reasoning, language, routing between models Nothing reasons; the system is a script AI platform team Planning Task decomposition, next action, stopping condition Agents loop without terminating and cost runs away AI platform team Memory Session state, long-term recall, retention policy Agents repeat work and forget corrections Data engineering Tool and action The registry of what the agent can physically do The agent talks but cannot act, or acts without limits Application and integration teams Knowledge and retrieval Indexes, freshness, grounding facts Confident answers from stale or missing data Data engineering Guardrail and policy Permissions, spend caps, approval checkpoints Excessive agency; one bad decision reaches production Security and risk Observability and evaluation Traces, cost, quality scoring over time Incidents cannot be explained and drift goes unnoticed Platform engineering
How the Architecture Runs: The Perceive, Reason, Act, and Observe Loop All seven layers cooperate in a loop that repeats until the task finishes or a limit stops it. One pass through that loop is a single agent turn.
One agent turn. The loop repeats until the task finishes or a limit stops it. The Four Steps of a Single Agent Turn Perceive. The agent receives a goal and assembles its working context. Session memory, relevant long-term memory, and whatever retrieval returns for the current question.
Reason. The planning layer decides what to do next. Sometimes that is a full plan. More often it is just the next single action, since committing to a long plan before seeing any results tends to age badly.
Act. Then the agent calls a tool through the tool layer, which enforces permissions and returns either a result or a refusal.
Observe. The result goes back into working context, the agent evaluates progress against the goal, and either loops or stops. Everything in the turn is written to the trace.
Where the Loop Breaks in Production Four failure points account for most of what goes wrong, and each one belongs to a specific layer.
Bad context at perceive. Retrieval returns something stale or irrelevant, and the agent reasons perfectly from a wrong premise. This is the most common failure and the least visible, because the output looks fine.Wrong tool at act. Two tools have similar descriptions and the model picks the wrong one. Usually a symptom of tool bloat.No validation at observe. A tool returns an error or an empty result, the agent treats it as a completed step, and the error compounds through the rest of the task.No stopping condition. The agent cannot finish, cannot recognise that it cannot finish, and keeps spending.Every one of those is an architecture gap, which is why swapping in a stronger model rarely fixes them.
Classical Agent Architectures and Where They Show Up in Modern Systems Long before language models, AI research had already classified agents by the structure of their decision function. Those five structures are still the cleanest vocabulary for deciding how much autonomy to grant, and they map directly onto the seven layers above.
The classical agent structures and their modern equivalents. A simple reflex agent maps the current input straight to an action with no memory and no model of the world. In a modern stack this is a single classify-and-route call. It is fast and cheap, although completely blind to anything outside the current input.
A model-based reflex agent keeps an internal representation of state, so it can act on things it cannot currently observe. That is the memory layer doing its job. In practice, it is what lets an agent know a ticket was already escalated an hour ago.
A goal-based agent chooses actions by reasoning about which ones move it toward a stated goal. That is the planning layer, and almost every system marketed as an AI agent today sits here.
Using the Taxonomy as an Autonomy Dial A utility-based agent goes further and weighs competing outcomes against a utility function. It picks the action that scores best on cost, speed, or risk. In practice this shows up as an explicit scoring step in the planning layer, and it is what you want when trade-offs are real and the agent must justify a choice.
A learning agent updates its own behaviour from feedback. In production this is almost never online weight updates. Instead, it is the evaluation layer feeding corrected examples back into prompts, routing rules, or a fine-tuning set on a schedule a human controls.
The practical use of this taxonomy is as an autonomy dial. Most failed agent programs reached for a goal-based or utility-based design where a model-based reflex agent would have been sufficient, auditable, and an order of magnitude cheaper to run. If you want the taxonomy from the product angle rather than the architectural one, see types of AI agents .
Single-Agent vs Multi-Agent Architecture Once the layers are settled, the next structural decision is how many agents the system runs and how they relate. This is a topology question, and the honest default is one.
When a Single Agent Is the Right Architecture A single-agent architecture has one reasoning engine, one tool registry, and one memory store. Consequently it is easier to debug, since there is one trace, easier to secure, since there is one permission set, and cheaper, since there is no inter-agent chatter to pay for.
It fits when the work is bounded and the tools are related.
Use case. Automating responses to standard customer inquiries such as order status, product details, and return policies.The same reasoning applies to document triage, data lookups, and report generation. If one competent person with access to a handful of systems could do the task, one agent can usually be architected to do it too. Our roundup of AI agent examples shows what that scope looks like in practice.
What Changes Structurally When You Add Agents A multi-agent architecture splits the work across several agents with distinct roles, and that split introduces three components a single agent never needs. A communication mechanism so agents can pass work and results. A shared state store so they do not each hold a private and divergent view of the task. And a coordination layer that decides who runs when.
Multi-agent designs come in two shapes, and the distinction decides where the coordination cost lands. A vertical topology puts one supervisor agent in charge, decomposing the goal and delegating to workers that report back to it. A horizontal topology has peers on a shared channel, each picking up work it recognises. Vertical is easier to audit because one agent owns the plan, and it is therefore the safer default in a regulated process. Horizontal handles open-ended work better and is much harder to trace, because no single record explains why the group settled on an answer.
Those three components are where multi-agent systems get expensive, in engineering time and in tokens. The architectural case for paying that cost is real but narrower than the marketing suggests. It holds when sub-tasks need genuinely different tools or permissions, when they can run in parallel, or when one agent’s context window cannot hold the whole job. Masterman and colleagues survey the field in The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling , and they treat single-agent and multi-agent as two design families with different patterns rather than as a ladder you climb.
For how agents actually divide and coordinate work, our guide to multi-agent AI systems covers the collaboration mechanics that sit on top of this structure, and AI agent orchestration covers running them at scale.
The Hybrid Workflow and Agent Architecture Most Enterprises Need The framing that serves most enterprises best is not single versus multi. It is how much of the process should be deterministic at all.
A purchase-order process has perhaps fifteen steps, and thirteen of them are rules. Validate the format, check the vendor exists, confirm the budget code, route by amount. Only two steps involve genuine ambiguity, for instance interpreting a non-standard line item or deciding whether an exception is reasonable.
Running all fifteen through an agent makes thirteen deterministic steps probabilistic for no benefit. The architecture that works puts a workflow engine in charge of the sequence and calls agent reasoning only at the ambiguous steps. As a result you keep the audit trail and the predictability of the workflow, while spending model cost only where judgement is actually required. Our look at AI agentic workflows covers where that line usually falls.
Kanerika Service
Agentic AI Design and Delivery
Kanerika designs agent architectures for enterprises already running Microsoft Fabric, Databricks, or Snowflake, and takes them from a scoped pilot to a governed production deployment.
Explore Agentic AI Services →
AI Agent Architecture Patterns, and the Two Vocabularies for Them Architecture patterns are the recurring shapes a working agent system takes. There are roughly five of them. The confusing part, though, is that the two most-cited sources give them completely different names.
Anthropic’s engineering guidance describes prompt chaining, routing, parallelization, orchestrator-workers, and evaluator-optimizer. Microsoft’s Azure Architecture Center describes sequential, concurrent, group chat with maker-checker loops, handoff, and magentic orchestration. An architect reading both documents in the same week can be forgiven for thinking they describe ten patterns. They largely describe the same five shapes from two vantage points, one framed around how the LLM is called and one framed around how multiple agents are coordinated.
Table 2. The same five shapes under both vendors’ names.
Shape Anthropic’s name Microsoft’s name Use it when Avoid it when Fixed order Prompt chaining Sequential orchestration The steps genuinely have a fixed order and each one improves the next Latency matters and the steps are independent Fan out Parallelization Concurrent orchestration Sub-tasks are independent, or you want several opinions on one input Steps depend on each other’s output Dispatch Routing Handoff orchestration Inputs fall into distinct classes that need different handling The classes overlap and misrouting is costly Plan and delegate Orchestrator-workers Magentic orchestration The sub-tasks cannot be known until the work starts The decomposition is already known and could be hard-coded Critique and revise Evaluator-optimizer Group chat with maker-checker Quality criteria are clear and a second pass measurably helps Quality is subjective, or the extra round-trip is not worth the latency
Where the Frameworks Sit in This Picture Frameworks implement these shapes, although they do not change them. LangGraph models an agent as an explicit state graph, which suits the plan-and-delegate and critique-and-revise shapes because the control flow is a first-class object you can inspect. LangChain sits a level lower as the plumbing for prompts, tools and retrieval. Microsoft’s Semantic Kernel and Azure AI Foundry Agent Service target teams already inside Azure identity and governance, which matters more than the API surface once an agent needs delegated permissions. CrewAI and AutoGen lean into the multi-agent case, with roles and a conversation between agents as the organising idea.
Pick the framework after the shape, never before it. After all, a framework decision is reversible in weeks, whereas a topology decision is reversible in quarters. Our side-by-side of CrewAI, AutoGen and the Microsoft Agent Framework compares them properly, and open source AI agents covers what you can run without a vendor contract.
Two things follow from reading the two vocabularies side by side. The pattern names in a vendor’s documentation describe that vendor’s SDK, not a standard, so a design decision recorded as “we chose handoff” ages badly once the team moves framework. Record the shape instead, because the shape is what survives a migration. And the augmented single agent, which Anthropic treats as the building block rather than a pattern, covers more production use cases than any of the five.
The Enterprise Reference Architecture for AI Agents Everything so far describes an agent. An enterprise agent architecture then adds the parts that exist because the agent runs inside an organisation with data it does not own, users it must authenticate, and auditors who will ask questions.
An enterprise AI agent reference architecture with the governance plane crossing every band. Where Enterprise Data Enters the Architecture Agents are only as good as the data layer beneath them, and in most enterprises that layer already exists. A lakehouse on Microsoft Fabric , Databricks , or Snowflake , plus operational systems reached through APIs.
The architectural decision is whether the agent queries those systems live or reads from a purpose-built index. Live queries are always fresh and always slower, and they put agent traffic onto systems sized for human traffic. An index, on the other hand, is fast but can be stale. Most production designs therefore use both, with an explicit freshness rule per data type, and that rule belongs in the architecture document instead of in someone’s head.
Identity, Permissions, and the Agent’s Own Credentials An agent acting on behalf of a user should not have more access than that user. Getting that right means the agent carries a delegated identity instead of a service account with broad rights, so that the same row-level security the human is subject to also applies to the agent’s query.
Service accounts are the shortcut almost every pilot takes, and almost every security review rejects them, because a compromised agent would then hold the union of everyone’s permissions. Agents also need an identity of their own, separate from any user, so that autonomous actions can be attributed. Our overview of agentic AI governance covers the policy side of that model.
Human Approval Checkpoints and the Audit Trail In practice, approval checkpoints are how an organisation buys autonomy gradually. The usual progression runs from a human approving every action, to approving only actions above a threshold, to spot-checking a sample. Each step is a configuration change instead of a rebuild, provided the checkpoint was designed in.
The audit trail is the other half, and it is the part auditors ask for first. For a regulated process, a reviewer will eventually ask which document the agent read, which rule it applied, and who approved the result. The NIST AI Risk Management Framework organises those obligations into govern, map, measure, and manage functions, and it is a reasonable structure for deciding what your trace has to capture before you write a line of it.
The Runtime: Where the Agent Actually Executes Agent runs are long, bursty, and stateful, which is therefore a poor fit for a request-response web tier. Sessions can last minutes, a single run may issue dozens of tool calls, and meanwhile the work has to survive a restart.
That pushes the runtime toward managed agent services or container orchestration with durable session state, rather than toward the function-as-a-service pattern teams reach for first. Getting this wrong tends to show up late, as timeouts and lost sessions under real concurrency, long after the pilot looked fine.
Securing an AI Agent Architecture Security for agents is mostly the discipline of limiting what the system can do, applied at design time instead of at review time.
Authenticate the agent as well as the user. In other words, every tool call should carry both identities, so an action can be traced to the person who triggered it and to the agent that performed it.
Scope permissions to the task, not to the agent. For example, an agent that handles refunds under a hundred dollars should hold a credential that cannot issue a larger one. Because that limit lives in the credential instead of the prompt, no phrasing can get around it.
Treat retrieved content as untrusted input. A document an agent retrieves can carry instructions aimed at the model. So if retrieved text can reach the planning layer and influence tool selection, the retrieval layer is an injection surface, and it needs the same treatment as user input.
Alert on the shape of activity. For instance, an agent making ten times its normal volume of calls is a signal, whether or not any single call failed.
Kanerika delivers this layer as governance services rather than as a product, through kanSuite. kanGovern for policy, kanComply for regulatory mapping, and kanGuard for runtime controls, built on the customer’s existing Microsoft Purview footprint where one exists.
On-Demand Webinar
The Real Cost of LLM Security Risks and How to Reduce Them
Where language model deployments actually leak, what the exposure costs, and the controls that close the gaps without stalling the programme.
Watch On Demand →
AI Agent Architecture vs RAG Architecture Retrieval-augmented generation and agent architectures get compared constantly, and the difference is cleaner than the debate suggests. RAG is a component. An agent is a system that usually contains one.
A RAG application retrieves relevant documents, passes them to a model, and returns an answer. Here the flow is fixed, and a human acts on the output. An agent chooses when to retrieve, what to retrieve, whether to retrieve again after reading the first result, and what to do with what it found.
Table 3. RAG architecture and agent architecture compared.
Dimension RAG architecture Agent architecture Goal Answer a question from a corpus Complete a task end to end Control flow Fixed, written by a developer Chosen at runtime by the model Tools Retrieval only Any registered action, including writes Memory Optional, often just the chat thread Required, split short-term and long-term Failure mode A wrong answer a person can check A wrong action already taken in a live system Testing Answer quality against known questions Behaviour scored across many runs
Why the Difference Changes the Approval Path That last row is the one that changes budgets. A wrong RAG answer is reviewed by a human before anything happens. A wrong agent action has already happened, and the remedy is a reversal rather than an edit. Teams that cost an agent programme on model spend alone consistently miss what the reversal path costs to build.
The practical consequence is that RAG and agent work sit on different approval tracks inside most enterprises. A retrieval assistant usually clears review as a search feature, because nothing it does changes a record. An agent with write access to the same corpus triggers a full change-control conversation, since the blast radius is no longer a bad paragraph on a screen. Knowing which track a use case belongs on, before the build starts, saves more calendar time than any framework choice. The comparison of RAG and agentic RAG covers the middle ground where retrieval itself becomes agentic.
Five Architecture Mistakes That Stall Agent Programs These are the five Kanerika’s teams find most often when a pilot is working and the production rollout is not.
Five architecture gaps that stall agent programs after a successful pilot. Starting from the model instead of the process. The first question is which decision in a business process is genuinely ambiguous. Teams that start from model selection build something impressive that no process owner asked for.Registering every tool you have. Tool bloat degrades selection accuracy well before it hits any technical limit. Group tools behind task-scoped interfaces and give the agent a short menu.Treating memory as free storage. Unbounded long-term memory slows retrieval, raises cost per call, and creates a data-protection obligation nobody scoped. Set a retention policy in the design, well before the first audit.Leaving observability for later. Traces are the only artefact that explains an agent’s behaviour, and they cannot be generated retrospectively for an incident that already happened.Granting autonomy before the controls exist. Approval checkpoints, spend caps, and scoped credentials are cheap to design in and expensive to add once an agent is live in a business process.Choosing an AI Agent Architecture: A Decision Framework Architecture choice follows from the shape of the work, not from the sophistication available. Start with the least autonomous design that solves the problem and add autonomy only when a specific limitation forces it.
Table 4. Matching the requirement to the architecture.
If the work looks like this Choose The layer that carries the risk A fixed process with two or three ambiguous steps Workflow engine calling agent reasoning at those steps Planning Questions answered from a document corpus RAG, no agent needed Knowledge and retrieval A bounded task across a handful of related systems Single agent with a scoped tool registry Tool and action Sub-tasks needing different tools, permissions, or parallelism Multi-agent with an explicit coordination layer Guardrail and policy Decisions with real trade-offs the business must justify Utility-based scoring inside the planning layer Observability and evaluation A regulated process with audit obligations Any of the above, plus approval checkpoints and a full trace Observability and evaluation
Checklist
Agentic AI Readiness Checklist
Work through the data, permission, and governance questions an agent architecture has to answer before a pilot is allowed near a production system.
Get the Checklist →
How Kanerika Designs Agent Architectures Kanerika builds agent systems for enterprises that already run a serious data platform, usually Microsoft Fabric, Databricks, or Snowflake. That starting point therefore shapes the method, because the hard part is rarely the agent and almost always the data, permission, and audit plumbing it has to sit inside.
The Four Stages of a Kanerika Agent Build The work runs in four stages.
Locate the ambiguity. Map the target process and find the specific decisions that are genuinely judgement calls. Everything else stays deterministic. This stage usually shrinks the scope, which is the point.Design the boundary before the agent. Define the tool registry, the credential model, the approval checkpoints, and the trace schema first. These are the constraints the rest of the build has to satisfy, and deciding them last is what produces an agent that cannot pass a security review.Build the context layer. Connect retrieval to the existing lakehouse and operational systems with an explicit freshness rule per data type. In our experience this stage determines agent accuracy more reliably than any model upgrade does.Instrument, then widen autonomy. Ship with every action approved and a complete trace, measure against a held-out task set, and remove approval steps only where the evidence supports it.The Reference Architecture, Running in Production A recent build shows the reference architecture in production. A global expert-network firm needed every expert vetted for negative news before an engagement, a process its compliance analysts ran by hand across news sites, social platforms, and professional records, then checked against a separate rulebook. Kanerika built an AI compliance agent that connects to internal databases to assemble the expert profile, runs rule-driven retrieval across those public sources, evaluates every finding against the client’s own disqualification criteria, and produces a structured report with citations that an analyst reviews. Each of the seven layers is visible in that description, and the result was 3X faster expert vetting, a 70% decrease in backlog cases, a 40% reduction in event delays, and negative news screening time down by 60%.
Case Study
3x Faster Expert Vetting with an AI Compliance Agent
How Kanerika replaced a manual negative-news screening process with a governed agent architecture, cutting backlog cases by 70% while keeping every finding reviewable by a human analyst.
Read the Case Study →
The Agents and Credentials Behind the Method Kanerika also runs several agents of its own in production, which is where a lot of this comes from. Karl for data insights, Klara for document compliance checks against a playbook, Mike for quantitative proofreading, and Susan for PII redaction. Kanerika is a Microsoft Solutions Partner for Data and AI with the Analytics Specialization, a Databricks Consulting Partner, a Snowflake Select Tier Partner, and an OpenAI Select Partner, and holds ISO 27001, ISO 27701:2019, ISO 9001:2015, SOC 2 Type II, and a CMMI Level 3 appraisal. Those certifications matter to this topic for a practical reason. The audit evidence an ISO 27001 or SOC 2 assessor asks for maps almost line for line onto what the observability layer has to capture, so a team that already runs those controls has most of the trace schema written before the agent exists.
Wrapping Up An AI agent architecture is a set of decisions about limits. What the agent can reach, what it can remember, what it is allowed to do, and what gets written down when it does something.
The model layer gets the attention, although it is the easiest of the seven to change. The other six are where agent programmes succeed or stall, and they are built the same way any dependable system is built, by deciding the boundary before the capability.
One practical test settles most design arguments. Ask what the agent is allowed to do that a workflow could not already do, and whether anyone would be able to explain the decision a week later. If the first answer is thin or the second is no, the architecture is not ready, whatever the demo looked like.
Start with the least autonomous design that solves the problem. Then add autonomy only when something specific forces it, and only once the trace exists to show what happened. When the design is settled, how to build AI agents covers the implementation, and agentic AI enterprise adoption covers the rollout.
Talk to Kanerika
Designing an Agent Architecture for a Real Process?
Bring the process and the data platform you already run. Kanerika’s architects will map where agent reasoning genuinely helps and what the permission and audit model has to look like before anything ships.
Book a Working Session →
Frequently Asked Questions
What is the architecture of an AI agent? An AI agent architecture is the set of components built around a language model so it can act. Seven layers do the work. The model reasons. A planner picks the next step. Memory holds state. Retrieval supplies facts. Tools execute. Guardrails set limits. Observability records what happened, so any run can be explained later.
What are the components of agent architecture? A model layer reasons. A planning layer decides the next action and when to stop. A memory layer holds session and long-term state. A tool layer executes against real systems. A retrieval layer supplies grounding facts. A guardrail layer sets hard limits. An observability layer records every step of the run.
What is a multi-agent architecture? A multi-agent architecture splits work across several agents with distinct roles. It adds three components a single agent never needs. A communication mechanism passes work and results between them. A shared state store keeps one consistent view of the task. A coordination layer decides which agent runs and when. Those three are where the engineering cost lands.
What are the main advantages of using a multi-agent architecture? Multi-agent designs help when sub-tasks need genuinely different tools or permissions. They also help when work can run in parallel, or when one context window cannot hold the whole job. Roles stay separate, so each agent carries narrower permissions. The cost is coordination overhead in engineering time and in tokens.
What are the common AI agent architecture patterns? Five shapes recur across production systems. A fixed order of steps. A fan-out of independent work. A dispatcher routing input to the right handler. A planner delegating to workers. A critique-and-revise pass. Anthropic and Microsoft publish these under different names, so the same shape often appears twice in vendor documentation.
What is the difference between RAG and AI agent architecture? RAG retrieves documents and passes them to a model along a flow a developer wrote in advance. An agent decides for itself when to retrieve, what to fetch, and what to do with the result. A wrong RAG answer gets reviewed. A wrong agent action has already happened. That difference sets how much testing each one needs.
What role does memory play in AI agent architecture? Memory splits into two structures that behave differently. Short-term memory holds the current task and should be discarded when the session ends. Long-term memory persists across sessions and behaves like a database, so it carries retention limits, deletion rights and access control. Unbounded memory raises cost and slows retrieval. Set a retention policy during design, not after the first audit.
How do AI agents use tools and APIs? Tools are registered in a tool layer that every agent action passes through. The model proposes an action and the tool layer decides whether that action is possible at all. Interfaces have been standardising on the Model Context Protocol, so a tool exposed once works with any protocol-aware agent. Permissions belong in that layer, never in the prompt.
How do you secure an AI agent architecture? Authenticate both the user and the agent on every tool call. Scope credentials to the task, so no phrasing in a prompt can widen them. Treat retrieved content as untrusted input, because a document can carry instructions aimed at the model. Alert on the shape of activity, not only failures.
What architecture is used for enterprise AI agents? Enterprise designs add four things to the standard layers. Delegated identity, so the agent inherits the user’s own permissions. A data layer built over the existing lakehouse or warehouse. Human approval checkpoints sized to the risk of each action. A governance plane carrying traces, audit logs and cost figures. That plane crosses every other layer in the stack.
What is an AI agent example? A compliance agent that vets outside experts before a client engagement. It assembles a profile from internal databases. It searches public news and professional sources under rule-based logic. It checks each finding against a disqualification rulebook. It produces a cited report that a human analyst reviews before anything gets approved.
What is the basic structure of an AI agent? The basic structure is a loop. The agent perceives its goal and assembles context. It reasons about the next action. It acts by calling a tool. It observes the result and decides whether to continue. Memory carries state between turns. A trace records each pass so the run stays explainable.
What does an AI agent architecture diagram look like? A useful diagram stacks the layers vertically and reads from the bottom up. Tools and retrieval sit at the base. Memory and the model sit in the middle. Planning sits above them. Guardrails and observability run across the top. Arrows show one request moving down through the stack and back.
How do AI agents work? An agent receives a goal, then gathers context from memory and retrieval. It asks the model what to do next. It calls a registered tool, reads the result, and repeats until the task is done or a limit stops it. Guardrails decide which actions are permitted at each step. Every step is written to a trace.
What is agent-based architecture? Agent-based architecture organises a system around autonomous components that pursue goals instead of following a fixed script. Each agent holds its own reasoning, tools and state. The design question is how much of the sequence a human fixes in code, and how much the agent is allowed to choose at runtime.
What is the architecture of an intelligent agent in AI? Classical AI describes five structures. A simple reflex agent maps input straight to action. A model-based reflex agent keeps internal state. A goal-based agent reasons toward an objective. A utility-based agent weighs competing outcomes. A learning agent updates itself from feedback. All five map onto the modern layered stack. The taxonomy still works as an autonomy dial.
What are the 5 types of agents in AI? The five classical types are simple reflex, model-based reflex, goal-based, utility-based and learning agents. They differ in how much state and reasoning the decision function carries. Most systems described as AI agents today are goal-based, with a planning layer choosing actions toward a stated objective and a memory layer holding state.
What is a single-agent architecture? A single-agent architecture uses one reasoning engine, one tool registry and one memory store to complete a task. It suits bounded work across a handful of related systems. Debugging is simpler because there is one trace. Security is simpler because there is one permission set. Cost stays lower too. Most bounded enterprise tasks fit this shape.
What are the advantages and limitations of using a single-agent architecture? The advantages are one trace to debug, one permission set to secure, lower cost and fewer moving parts. The limits appear when sub-tasks need genuinely different tools or permissions. They also appear when steps could usefully run in parallel, or when the job outgrows one context window. Start here regardless.
