TL;DR
AI agent infrastructure is everything you must run to keep agents working in production, beyond the model itself. It covers the runtime that hosts the agent loop, the stores that hold state and memory, and the gateways that connect tools. It also includes sandboxes for code, queues for long jobs, a secrets store, observability and cost metering. Many pilot failures come from this layer, such as timeouts, lost sessions, model quotas and runaway bills. Managed agent platforms remove much of the operations work, while self-hosting buys control at the price of more engineering. Size the stack from concurrent sessions and task duration, then load test before you scale.
Key Takeaways AI agent infrastructure is the runtime, state, memory, tool, sandbox, queue, secrets and observability services that host an agent, not the model. Agent work runs for minutes or hours and calls many tools, so ordinary request-response hosting breaks first. State must live outside the process, because a restart or a scale-in event erases anything held in memory. Model quotas and downstream API limits cap throughput long before CPU does, so sizing starts with concurrency. Cost per successful task, including retries and abandoned sessions, is the number that keeps the bill honest. Managed platforms suit standard workloads, self-hosting suits strict control needs, and many enterprises end up hybrid. Watch on YouTube
Why AI Agents Fail in Production
A Kanerika take on Luna, the AI agent that helped a boutique lose $13,000, and the missing scope, boundaries and human approval behind it.
What Breaks First When a Pilot Agent Meets Real Traffic Imagine a team that has piloted an internal research agent for six weeks. Forty testers loved it, so at 9 a.m. on a Monday it opens to two thousand employees. At 9:20, the first research reports start dying halfway through, because the longest tasks are hitting the function timeout.
By ten, the model endpoint is also returning rate-limit errors, because every user now shares one tokens-per-minute quota. Then a routine pod restart wipes every conversation that lived in process memory. On Friday, finance opens the invoice and finds retries billed twice and idle sessions billed all week.
Every one of those failures sits below the prompt. They live in AI agent infrastructure, the layer that hosts, remembers, connects and meters the agent. The stakes are also real. Gartner expects over 40% of agentic AI projects to be canceled by the end of 2027 , and escalating costs head its list of reasons.
What Is AI Agent Infrastructure? AI agent infrastructure is the set of runtime, storage, connection and operations services that keep deployed agents running. It hosts the agent loop, saves its progress, gives it safe access to tools and data, and records what it did and what it cost. The model sits at the center, yet it is usually the one piece you rent rather than run.
Still, the term often gets mixed up with design. How an agent plans, reflects and hands work to other agents is a question of AI agent architecture , which covers logical layers and patterns. Infrastructure is the physical answer to where each of those layers executes, how it scales and what happens when it fails.
Agent Infrastructure vs Model and GPU Infrastructure Model infrastructure serves tokens. That means GPUs, inference servers and the capacity planning behind them, which our guide to AI workloads and their compute needs covers in depth. Agent infrastructure sits one level up, because it consumes those tokens through an API and wraps them in long-running work.
So a team can run serious agents without owning a single GPU. Instead, their agent workers are often plain CPU containers that call a hosted model, a database, a queue and a dozen business APIs. The hard problems are duration, state and fan-out, rather than raw compute.
Why Agent Workloads Strain Ordinary Application Hosting A typical web request finishes in milliseconds and forgets everything when it ends. An agent task behaves differently. It may run for twenty minutes, call fifteen tools, wait on a human approval and then resume the next morning.
As a result, that profile breaks several assumptions at once. Load balancers expect short connections, serverless platforms cap execution time, and autoscalers assume each request costs roughly the same. Because agentic AI decides its own next step, two identical prompts can also produce very different bills.
The AI Agent Infrastructure Stack at a Glance Most agentic infrastructure resolves into ten services. Some are optional for a pilot, but almost all become mandatory once real users and real money are involved. The table below works as an inventory you can check your own platform against.
Component What it runs or stores Typical technology Needed in a pilot? Agent runtime The agent loop and its sessions Managed agent runtime, containers, serverless functions Yes Model access layer Calls to hosted models, quotas, routing, fallbacks Model API, LLM gateway Yes State store Checkpoints, task status, session IDs PostgreSQL, Azure Cosmos DB, durable workflow engine Yes, even if small Memory and retrieval store Conversation history, long-term memory, indexed knowledge Redis, vector database, search index Often Tool and MCP gateway Routes tool calls to APIs and MCP servers API gateway, MCP gateway Optional Execution sandbox Agent-written code, file work, browser sessions Containers, microVMs Only if the agent runs code Task queue Buffers background work, absorbs spikes Azure Service Bus, Amazon SQS, Kafka Optional Secrets store API keys, tokens, runtime configuration Azure Key Vault, AWS Secrets Manager Yes Observability Logs, traces, metrics for every run OpenTelemetry, Azure Monitor, CloudWatch Basic logs Cost metering Spend per agent, team and task Cloud cost tools plus token tagging Rarely, which is the mistake
Read the table from top to bottom and a pattern appears. The upper rows keep the agent alive, the middle rows let it act, and the bottom rows tell you whether it is worth keeping. Our enterprise AI architecture guide shows where this stack sits inside a wider platform.
What a Pilot Needs vs What Production Needs A pilot can survive on a runtime, a model key, a small database and log files. That is also why demos look cheap. Ten friendly testers rarely run tasks in parallel, rarely abandon a session halfway and never hit a quota.
Production adds the services that handle concurrency and failure. First, queues appear because traffic arrives in bursts. Sandboxes appear once the agent writes code, and cost metering appears the first time someone asks which team spent the budget.
The safest plan is to design for all ten services on day one, then switch on only what the pilot needs. Retrofitting state storage or tracing into a live agent is far more painful than leaving a slot for it.
Agent Runtime and Compute: Where the Loop Actually Runs The runtime is the process that executes the agent loop. It receives a task, calls the model, runs tools, writes state and repeats until the task is done. Three hosting models dominate, and each fails in a different way.
Serverless Functions, Containers and Managed Agent Runtimes Serverless functions are the fastest way to start, but they have hard ceilings. An AWS Lambda function can run for at most 900 seconds , so it suits a quick lookup agent but kills a forty-minute research task.
Containers, by contrast, remove the ceiling and keep costs elastic. Azure Container Apps scales on KEDA rules such as HTTP traffic or queue depth, and its default minimum is zero replicas. That makes it a natural home for agent workers that sleep between bursts.
Managed agent runtimes then go further. In Amazon Bedrock AgentCore Runtime , each user session gets its own microVM and can run for up to eight hours. Similarly, Microsoft Foundry Agent Service runs a hosted runtime on Azure, covered in the managed versus self-hosted section below.
Hosting option Run-time limit Scale to zero Best fit Serverless functions Hard cap (15 minutes on Lambda) Yes Short, bursty, single-step agents Containers with event scaling Set by you Yes, with cold starts Queue-driven workers, most production agents Managed agent runtime Platform limit (up to 8 hours per session on AgentCore microVMs) Platform-managed Teams that want session isolation without running a cluster Kubernetes or VMs you operate None beyond your own settings Only with extra tooling Strict data residency, custom runtimes, steady high volume
Stateless Workers vs Long-Lived Sessions Stateless workers pick up a task, load its state from a store, do one step and write the result back. Any worker can continue any task, so scaling and recovery are simple. The trade-off, however, is extra reads and writes on every step.
Long-lived sessions keep the agent in one process for the whole conversation. They feel faster and suit interactive copilots, but a crash loses everything not yet saved. In practice, most production designs use stateless workers for background tasks and short-lived sessions, with frequent checkpoints, for chat.
Cold Starts, Idle Sessions and Dropped Connections Three small runtime details cause outsized pain. Cold starts add seconds when a scaled-to-zero worker wakes, which users notice in chat but not in batch work. Idle sessions hold memory and sometimes paid compute while a user has gone to lunch.
Dropped connections are the subtle one, though. When a browser tab closes mid-task, the agent should keep working and post the result somewhere the user can find it later. That only works when the task runs on a worker behind a queue instead of inside the HTTP request that started it.
Set an idle timeout for every session type, and decide up front which tasks survive a disconnect. Those two settings remove most of the waste and most of the “my report vanished” tickets.
The Model Access Layer: Endpoints, Quotas and Routing Every agent step calls a model, so the path to the model is shared infrastructure. Providers meter that path tightly. Azure OpenAI quotas are set in requests per minute and tokens per minute per model and deployment type, and other providers work the same way.
In practice, that quota is the first wall most agent platforms hit. One busy agent that loops through a long document can starve every other agent on the same deployment. So production teams put a model gateway in front of the endpoints. The gateway shares quota fairly, retries with backoff and then falls back to a second region or model.
The gateway is also where routing pays off. Simple classification steps can go to a small, cheap model, while planning steps go to a stronger one. Our guide to LLM architecture explains the model side of that choice, and the enterprise LLM guide covers deployment options.
Do AI Agents Need GPUs? Usually not. When the model runs behind a hosted API, the agent workers are ordinary CPU processes that spend most of their time waiting on network calls. GPUs enter the picture only when you self-host the model or run heavy local steps such as embedding large document sets.
So the distinction keeps budgets sane. Put GPU spend in the model tier’s budget, and size the agent tier on CPU, memory and concurrent sessions instead.
State and Memory Stores: Keeping Work Alive Past a Restart An agent that forgets its progress on every restart cannot run in production. Containers get rescheduled, nodes get patched and deployments roll out several times a week. Each of those events kills any task whose progress lives only in process memory.
The fix, therefore, is to treat state as data with its own home. Execution progress, conversation history and long-term knowledge each need a different store, and mixing them creates the slow, expensive designs that pilots drift into.
Execution State and Checkpoints After a crash, execution state tells the platform where a task was and what it had already done. The agent writes a checkpoint after each meaningful step, with the task ID, the step number, the tool results so far and any pending approval.
Agent frameworks provide this as a feature, but the store behind it is your responsibility. LangGraph’s persistence layer separates short-term checkpoints from long-term stores, and its in-memory saver does not survive a restart. In production, that checkpointer has to point at a real database such as PostgreSQL.
Durable workflow engines take the idea even further. Temporal records an event history for every workflow , so a failed run resumes from the last recorded event instead of starting over. For agents with long approval waits, that guarantee is worth the extra component.
Short-Term Context, Long-Term Memory and Retrieval Short-term context is the working set for one conversation, such as recent messages and tool outputs. It changes on every turn and needs fast reads, so an in-memory store like Redis or the checkpoint table itself fits well. It should also expire, because last month’s chat is noise in today’s prompt.
Long-term memory, by contrast, holds what the agent should remember across sessions, such as user preferences, past decisions or facts learned from earlier tasks. Retrieval stores hold enterprise knowledge the agent reads but does not write, usually documents split into chunks and indexed for semantic search. Both power the patterns described in our guides to the RAG pipeline and agentic RAG .
Choosing the Store for Each Kind of State The table below maps each kind of agent state to a sensible default. Treat it as a starting point, then adjust for the databases your platform team already runs well.
State type Typical lifetime Sensible default store What to watch Checkpoints and task status Hours to days PostgreSQL, Azure Cosmos DB, workflow engine Write volume per step, unbounded growth Short-term conversation context Minutes to hours Redis or the checkpoint store Expiry rules, memory pressure Long-term agent memory Months Relational table plus vector index Stale facts, deletion requests Enterprise knowledge for retrieval Until source changes Vector database or search index Re-indexing lag, access filtering Files and artifacts Task lifetime or longer Object storage Orphaned files after failed tasks
Vector databases deserve a careful pick because they are hard to swap later. Our vector database comparison sets the main options side by side on scale, filtering and cost.
Retention, Cleanup and Recovery Agent storage grows steadily, even when traffic is flat. Every step writes a checkpoint, every session leaves history and every failed task can leave temporary files behind. Without expiry rules, the state store becomes the largest database in the platform within a quarter.
Set a retention period for each state type, and run a cleanup job that removes orphaned sessions and files. Back up the checkpoint store like any production database, and test a restore at least once. A recovery plan you have never run tends to fail on the day it matters.
Tool Gateways, MCP Servers and Execution Sandboxes Tools are how an agent acts, so they reach well beyond your runtime. Each tool call leaves your runtime and hits a CRM, an ERP, a search index or a code interpreter. At pilot scale, the agent simply calls those systems directly. At production scale, that sprawl becomes impossible to monitor or govern.
What a Tool Gateway Does To contain that sprawl, a tool gateway puts one front door in front of every tool. Agents call the gateway, and the gateway handles routing, authentication, timeouts, retries and logging. When an API changes or goes down, you fix it in one place rather than in every agent.
Managed platforms now also ship this as a service. Amazon Bedrock AgentCore Gateway converts existing APIs and functions into MCP-compatible tools and connects to MCP servers you already run. Self-hosted teams usually build the same thing on an API gateway plus a small registry of approved tools.
Whether the gateway should block a risky call is a policy question, and it belongs to AI guardrails that run at the tool boundary. Infrastructure’s job is to make sure every call passes through a point where that policy can run.
Hosting MCP Servers: Transports and Sessions The Model Context Protocol has become the common way to expose tools to agents. How you host an MCP server affects scaling more than most teams expect. Local servers talk over stdio and live on the same machine as the agent, which is simple but hard to share across a fleet.
Remote servers use the Streamable HTTP transport, which replaced the older HTTP plus SSE transport. Earlier revisions let a server pin a client to one instance through an Mcp-Session-Id header, so the load balancer had to route that client back to the same place. The 2026-07-28 MCP specification removes protocol-level sessions, and servers that need state across calls now pass explicit handles as tool arguments. That state still has to live in a shared store, and servers built on older revisions keep their sessions until they are upgraded.
The operational rule is simple. Prefer stateless MCP servers wherever the tool allows it, because they scale like any web service. Keep stateful ones few, give them sticky routing and plan for their sessions to expire.
Watch on YouTube
MCP’s Biggest Update Yet: Moving AI Agents Toward Production
Kanerika explains what the latest Model Context Protocol changes mean for running agents in production, including how tool servers are hosted and connected.
Sandboxes for Code and Browser Work Some agents write and run code, fill in web forms or process uploaded files. That work should never run inside the agent’s own process, because one bad script can take the agent down. A sandbox gives each task a short-lived, isolated environment with its own filesystem, CPU and memory limits, then throws it away.
Fortunately, microVMs have made this cheap enough for routine use. Firecracker boots a microVM in under 125 ms with less than 5 MiB of overhead, which is why several agent platforms build on it. Containers with strict limits work too, provided each task gets a fresh one.
Even so, sandboxes still cost capacity. Budget for peak concurrent sandboxes, cap how long each may run and clean up temporary storage when a task ends. An unbounded sandbox pool is one of the quickest ways to turn a modest agent into a large invoice.
Queues, Background Jobs and Failure Recovery Agent traffic arrives in bursts, while the systems agents depend on have fixed limits. A queue sits between the two. It accepts work as fast as users submit it and hands it to workers only as fast as the platform can safely process it.
Why a Queue Belongs Between Request and Work Without a queue, every user request holds a connection open until the agent finishes. Long tasks time out, spikes overload the model quota and a single slow tool backs up the whole service. With a queue, the request returns a task ID at once, and a worker picks the job up when capacity frees.
The same pattern also supports status tracking. Clients poll or subscribe for progress, so a dropped connection no longer kills the task. Patterns for coordinating many agents on top of this layer are covered in our guide to AI agent orchestration .
Retries, Dead-Letter Queues and Duplicate Work Queues trade one problem for another, because most of them deliver a message at least once rather than exactly once. Amazon SQS standard queues state plainly that more than one copy of a message might arrive. For an agent, a duplicate can mean two emails sent, two tickets opened or two payments queued.
The main defence, then, is idempotency. Give every task and every side-effecting tool call a unique key, and have the tool or the gateway refuse a key it has already processed. Then a retry repeats the thinking but not the action.
Retries also need a limit, though. An SQS redrive policy sets maxReceiveCount , after which a failing message moves to a dead-letter queue. Someone must own that queue, because every message in it is a task a user is still waiting on.
Backpressure and Downstream Limits Adding workers, however, does not add capacity to the systems they call. If your ERP accepts fifty requests a second, two hundred agent workers will just get throttled faster. Worker concurrency, queue depth and tool rate limits must be set together.
For that reason, watch queue depth as a leading signal. When it grows steadily, either the workers are too few or a downstream system is slowing, and the traces will show which one.
Secrets, Configuration and Release Operations Agents hold more credentials than most applications, since every tool needs its own access. How those credentials are stored, delivered and rotated is an infrastructure task. Which permissions each agent should have is a security design question, covered in our guide to agentic AI security .
Credentials as a Platform Service Keep every key in a central secrets store such as Azure Key Vault or AWS Secrets Manager. Then inject it at runtime through a managed identity. Also, never bake keys into container images or prompt templates. Rotation then becomes a scheduled job rather than a redeploy of every agent.
Prefer short-lived tokens for tool access where the target system supports them. When a token leaks, its useful life is minutes rather than months.
Environments, Versions and Rollbacks An agent release changes more than code. A new prompt, a new model version or a new tool schema can each change behavior, so all of them need versioning. Keep separate development, test and production environments, each with its own state store and model deployment.
Ship with infrastructure as code and staged rollouts, starting with a small share of traffic. Keep the previous version deployable for a quick rollback. Long-running tasks complicate this, because a task that started on version one may finish on version two, so record the version in every checkpoint.
Observability for AI Agent Infrastructure Logs alone cannot explain an agent failure, because one task is spread across many services. A single task may touch the model ten times, call six tools and pass through three workers. You need traces that tie all of it together, plus metrics that warn you before users do.
Traces That Follow One Task End to End Give every task a trace ID that travels through the queue, the worker, the model gateway and every tool call. Record each step as a span with timing, token counts and status. OpenTelemetry’s generative AI semantic conventions now include agent spans and MCP conventions, so standard tooling can read them.
That trace answers the two questions every incident starts with, namely where the time went and which step failed. Our guide to AI agent observability covers the quality side, such as judging whether an answer was right. Our LLM observability guide covers model-level signals.
Kanerika Service
Keep Production Models Reliable With MLOps Consulting
Kanerika sets up the CI/CD release pipelines, monitoring and alerting, and drift detection that keep production models dependable after launch.
Explore MLOps Consulting Infrastructure Metrics Worth Alerting On In practice, most agent dashboards track the wrong things first. Token counts are useful, but the metrics below are the ones that predict an outage or a budget overrun.
Area Metric Why it matters Runtime Concurrent sessions, CPU and memory use, restarts Shows saturation before tasks start failing Model access Rate-limit errors, p95 model latency, fallback rate Quota is usually the first ceiling Tools Error rate and p95 latency per tool One slow API stalls every agent that uses it Queues Depth, age of oldest message, dead-letter count Early warning of backlog and stuck work Tasks Completion rate, duration, timeout frequency The user-facing reliability number Cost Spend per successful task, per agent and per team Catches runaway loops within hours, not at month end
Runbooks for the Failures You Will Actually See Write runbooks before launch, because the same failures recur. In most agent platforms, the list is short and predictable.
Model quota exhausted, so shift traffic to a fallback deployment and pause low-priority agents. A tool API is down, so open its circuit breaker at the gateway and queue affected tasks for retry. The dead-letter queue is growing, so inspect a sample, fix the cause and redrive. An agent is looping, so kill the task at its step or cost budget and alert the owner. The state store is filling up, so run cleanup and check retention settings. Each runbook also needs an owner and an alert that points to it. Measuring whether agents give good answers is a separate discipline, covered in our guide to AI agent evaluation .
Checklist
Enterprise AI Checklist
A practical checklist for taking AI from pilot to production, covering data readiness, platform choices, operations and ownership.
Get the Checklist → What Does AI Agent Infrastructure Cost? Token prices get the headlines, but they are only one line of an agent’s bill. Runtime hours, sandboxes, storage, gateways, tracing and the engineers who run them all add up. Teams that budget on tokens alone tend to discover the rest in month two.
Fixed vs Usage-Based Costs Fixed costs run whether or not anyone uses the agent. They include always-on runtime capacity, database instances, gateways, monitoring retention and the platform team’s time. Usage-based costs scale with work, such as model tokens, sandbox minutes, tool API fees and queue operations.
The split matters for pricing internal chargebacks. A low-volume agent can look cheap per token and still be expensive per task, because its fixed share is spread across very few runs.
Cost Drivers Beyond Tokens Several hidden drivers show up again and again in agent bills. Long sessions keep compute warm while users think. Retries repeat model calls, and loops that never converge can burn a day’s budget in an hour. Trace and log retention also grows with every step the agent records.
Context size, meanwhile, is the quiet one. Agents resend instructions, tool definitions and history on every step, so input tokens dominate. Prompt caching helps here. For instance, Anthropic bills cache reads at a tenth of the base input price on most models, and less on its newest ones, while five-minute cache writes cost 25% more than normal input.
Estimating Cost per Successful Task Measure cost per successful task, not per request or per token. Failed and abandoned tasks still cost money, so dividing by completed work gives the honest figure.
cost per successful task =
( model tokens
+ runtime compute
+ sandbox minutes
+ tool and API fees
+ share of storage, queues and monitoring )
/ tasks that completed successfullyHere is an illustrative example with assumed numbers. Say an agent starts 10,000 tasks a month and the total bill is $4,000, which is $0.40 per started task. If only 8,500 tasks complete, the real cost is about $0.47 per successful task, roughly 18% higher than the first figure suggested.
That gap is the budget for improving reliability. Every point of completion rate you recover lowers the unit cost without touching the price of a single token.
Cutting Waste Without Cutting Reliability Begin with the waste that has no upside. Set idle timeouts, cap steps and spend per task, and shut sandboxes down as soon as work ends. Route simple steps to smaller models and cache stable prompt prefixes.
Then attribute every cost to an agent and a team, using tags on cloud resources and labels on model calls. Our guide to cloud cost management also covers the tagging and reporting side. Once each team sees its own number, the most expensive agents tend to get attention first.
Managed vs Self-Hosted AI Agent Infrastructure Every team eventually asks whether to run the stack itself or rent it as agentic AI infrastructure as a service. The honest answer depends on how standard your workload is and how much control your compliance team demands.
Managed Agent Platforms Managed platforms run the runtime, scaling and much of the plumbing for you. Microsoft Foundry Agent Service hosts prompt agents and containerized hosted agents, and manages conversations, tool calls and lifecycle. It also lets you bring your own storage, Azure AI Search and Azure Cosmos DB for conversation state.
Amazon Bedrock AgentCore takes a modular route instead. Its runtime, memory, gateway, browser, code interpreter and observability are separate services you can adopt one at a time. Our Azure AI Foundry guide covers the Microsoft side. When the cloud itself is still open, our AWS vs Azure vs Google Cloud comparison also helps.
Self-Hosted on Containers and Kubernetes Self-hosting means running agent workers on your own containers, Kubernetes clusters or virtual machines, with databases, queues and gateways you operate. You pick any framework and any model, and you control exactly where data lives.
The price, though, is engineering time. Your team owns scaling, patching, session isolation, sandboxing and on-call. Open frameworks make the agent code portable, as our review of AI agent frameworks shows, but the infrastructure around them is still yours to build.
Decision factor Managed agent platform Self-hosted stack Time to first production agent Weeks Usually longer, since the platform must be built first Runtime operations Mostly the provider Your platform team Scaling and session isolation Built in, within platform limits Designed and tuned by you Control over data location and runtime Bounded by regions and features offered Full Cost shape Service fees plus usage Compute and storage plus staff time Compliance evidence Provider reports such as SOC 2, plus your own controls Entirely your own audit scope Portability Tied to the platform’s APIs Depends on the components you choose
Hybrid: Managed Runtime, Your Data and Tools In the end, many enterprises land in the middle. They use a managed runtime for the agent loop and keep state stores, knowledge indexes and tool APIs inside their own network. That keeps sensitive data where audit teams expect it while removing the hardest operations work.
A workable rule is to favor managed runtimes when the standard features meet your needs. Self-host only the pieces where residency, a custom runtime or steady high volume justify the extra work.
Case Study
82% Less Manual Processing With Cloud-Ready LLM Infrastructure
Kanerika upgraded a real estate developer’s infrastructure for GCP readiness and built LLM processing for vendor agreements, raising cloud integration efficiency by 75%.
Read the Case Study → How to Size AI Agent Infrastructure for Production Sizing an agent platform starts with work, not with servers. You need to know how many tasks arrive, how long each runs and how much each one asks of the model and the tools.
Estimate Demand Before Provisioning First, collect five numbers from the pilot logs. They are peak tasks per hour, average task duration, model calls per task, tokens per call and tool calls per task. If the pilot was small, multiply by the expected user count and add a margin for the novelty spike that follows any launch.
Concurrency With Little’s Law Little’s Law turns those numbers into concurrency. The average number of tasks in flight equals the arrival rate multiplied by the average time each task takes.
concurrent tasks = arrival rate x average duration
= 10 tasks per minute x 6 minutes
= 60 tasks in flight
worker slots = 60 x 1.5 headroom = 90
model tokens/min = 10 tasks x 8 calls x 3,000 tokens
= 240,000 tokens per minute at peakThe example uses assumed numbers, but the method holds. Sixty tasks in flight need ninety worker slots once you add headroom, and peak traffic needs 240,000 tokens per minute. If the model quota is lower than that, no amount of extra compute will help.
Size Stores, Queues and Tool Connections Next, apply the same arithmetic to every dependency. Checkpoint writes per minute equal tasks in flight times steps per minute. Tool calls per second at peak must fit under each downstream API’s limit. Sandbox capacity must cover the share of tasks that run code, multiplied by how long each sandbox lives.
Plan queue capacity for the worst hour, not the average one. A queue that holds a morning backlog without losing messages is cheaper than an outage.
Load Test Before You Scale Load tests for agents need realistic task mixes, because a test that sends one short prompt a thousand times proves very little. Run burst tests, long-session tests and restart tests, where you kill workers mid-task and confirm that the work resumes.
Track p95 task duration, completion rate and cost per successful task during each test. When any of them degrades before your target load, you have found the next ceiling.
Scaling AI Agent Infrastructure From Pilot to Production Few teams build the full stack in one go, and they should not try. A staged path adds each service when the workload needs it, while keeping the design ready for the next stage. Our guide to running an AI pilot covers how to pick the first use case.
Watch on YouTube
Why Do Enterprise AI Pilots Fail After the Demo?
A short look at why enterprise AI pilots that impress in a demo lose momentum before production, and what teams change to get them over the line.
Stage 1: A Working Pilot Deploy one agent on a managed runtime or a single container service. Persist state in a real database from the start, even if it is small, and keep basic logs. The goal is to learn task duration, tool usage and cost per task from real users. Our guide to building custom AI agents also covers this stage.
Stage 2: Production Support Services Add the queue, retries with idempotency keys, a dead-letter queue and end-to-end tracing. Then move secrets into a central store and split environments. At this stage, the agent should survive a worker restart without losing work.
Stage 3: Ready for Concurrency Introduce autoscaling on queue depth, a model gateway with fallbacks and per-task budgets for steps and spend. Run the load tests described above and set alerts on the metrics that matter. Cost metering by agent and team belongs here, before usage grows.
Stage 4: Many Agents, Many Teams Once several teams ship agents, shared services pay off. Standardize the runtime, the tool gateway and the observability stack, and give each team quotas and a cost report. Multi-agent AI systems also arrive at this stage, along with the ownership and approval questions covered in our agentic AI governance guide.
Production Readiness Checklist Before you open an agent to a large audience, confirm each item below.
The runtime supports the longest expected task and the peak concurrent sessions. Execution state survives a worker restart, and you have tested it. Memory stores have retention, cleanup and backup settings. Model quota covers peak tokens per minute, with a fallback route. Tool and MCP endpoints have measured availability and rate limits. Sandboxes have time limits, resource caps and automatic cleanup. Background jobs use idempotency keys, retry limits and a monitored dead-letter queue. Secrets live in a central store and rotate on a schedule. Traces, metrics and alerts cover every component, each with a runbook. Cost per successful task is reported per agent and per team. Every release is versioned and can be rolled back. How Kanerika Builds and Runs AI Agents Kanerika builds and runs AI agents for enterprise clients. Its agentic AI services cover the full lifecycle, from strategy and architecture to orchestration and production deployment.
Agents are built one workflow at a time, in four steps. First the team picks the workflow, ranking candidates by volume, clear rules and cost of errors. Then it designs the agent, defining the tools it can call, what it remembers and where a person approves before it acts. Next, the agent gets scoped access to ERP, CRM and documents and is scored on historical cases before go-live. Finally, monitoring, audit logs and a review loop keep accuracy up in production.
Kanerika’s MLOps consulting practice covers the CI/CD, monitoring and alerting around deployed models. The sizing method and load tests in this guide fit naturally into the design and test steps.
Kanerika is a Microsoft Solutions Partner for Data and AI and an OpenAI Select Partner. Kanerika also builds on both Anthropic’s Claude and OpenAI’s models. As a result, a model gateway can route each step to the right model instead of locking a client to one vendor.
Results From Kanerika AI Deployments The results show up in delivery. For a global expert network whose matching process spanned three disconnected systems, Kanerika built a context-aware AI agent that matches experts to surveys using semantic search. The client saw an 80% decrease in mismatch tickets, a 40% increase in mapping accuracy and 22% bandwidth savings for its team.
Case Study
80% Fewer Mismatch Tickets With a Context-Aware AI Agent
Kanerika built a context-aware AI agent that matches experts to surveys with semantic search, lifting mapping accuracy by 40% for a global expert network.
Read the Case Study → For a real estate developer, Kanerika upgraded the client’s infrastructure for GCP readiness and built LLM-based processing for vendor agreements stored as unstructured PDFs. Manual processing time fell by 82%, cloud integration efficiency rose by 75% and vendor selection improved by 90%.
Three pitfalls from this guide are worth checking first. Teams keep state in memory for too long, and they size for CPU instead of model quota. They also measure cost per token rather than cost per successful task. Fixing those three early tends to pay off faster than tuning prompts or models later.
Wrapping Up AI agent infrastructure decides whether a promising pilot becomes a dependable service or a cancelled project. The model matters, yet production agents fail more often on timeouts, lost state, exhausted quotas, duplicate actions and unmetered spend. Build the ten services deliberately, keep state outside the process and size from concurrency and quota. Pick managed, self-hosted or hybrid per component, and track cost per successful task from the first week. Teams that do this ship agents that survive Monday morning and still make sense on the Friday invoice.
Frequently Asked Questions
What is AI agent infrastructure? AI agent infrastructure is the set of services that host and operate AI agents in production. It includes the runtime that executes the agent loop, stores for state and memory, gateways for tools and MCP servers, sandboxes for code, queues for long jobs, a secrets store, observability and cost metering. The model itself is usually rented through an API.
What infrastructure do AI agents need to run in production? At minimum, an agent needs a runtime that supports its longest task, a model endpoint with enough quota, a durable state store and basic logging. Production use adds a task queue with retries, a tool gateway, sandboxes if the agent runs code, a central secrets store, end-to-end tracing and cost reporting per agent and team.
How is AI agent infrastructure different from LLM infrastructure? LLM infrastructure serves the model, which means GPUs, inference servers and capacity planning for tokens. AI agent infrastructure sits above it and runs the agent code that calls the model. It manages long-running tasks, saved state, tool access, queues and monitoring. Many teams run agents on CPU containers while renting the model as a hosted API.
Where should AI agents be hosted? Short, single-step agents fit serverless functions, as long as tasks finish inside the platform’s time limit. Most production agents run as queue-driven workers on containers. Managed agent runtimes such as Microsoft Foundry Agent Service or Amazon Bedrock AgentCore suit teams that want session isolation without operating a cluster. Kubernetes or virtual machines suit strict residency or custom runtimes.
Do AI agents require GPUs? Usually not. When the model is called through a hosted API, agent workers are ordinary CPU processes that spend most of their time waiting on network calls to the model and tools. GPUs are needed only when you self-host the model or run heavy local steps, such as embedding very large document collections, inside the agent platform.
What database is best for AI agent memory? No single database fits every kind of agent state. Checkpoints and task status suit a relational database such as PostgreSQL or a document store such as Azure Cosmos DB. Short-term context fits Redis. Long-term memory and enterprise knowledge usually combine a relational table with a vector index, while files and artifacts belong in object storage.
What is an MCP gateway in AI agent infrastructure? An MCP gateway is a single entry point between agents and the tools they use through the Model Context Protocol. Agents call the gateway, which routes requests to the right MCP server or API and handles authentication, timeouts, retries and logging. It gives the platform one place to monitor tool use, apply policy and fix broken integrations.
Why do AI agents need sandboxes? Agents that write and run code, browse websites or process uploaded files should not do that work inside their own process. A sandbox gives each task a short-lived, isolated environment with its own filesystem and resource limits, then discards it. That isolation protects the host, stops tasks from interfering with each other and keeps leftover files from piling up.
How much does AI agent infrastructure cost? Cost depends on volume, task length and hosting choice, so there is no standard price. The bill combines model tokens, runtime compute, sandbox minutes, tool and API fees, storage, queues, monitoring and staff time. The most useful measure is cost per successful task, because failed, retried and abandoned tasks still consume budget without delivering any result.
How do you control AI agent infrastructure costs? Start by measuring cost per successful task for each agent and team. Then set idle timeouts, cap steps and spend per task, and shut sandboxes down when work ends. Route simple steps to smaller models, cache stable prompt prefixes and tag every resource so each team sees its own share of the bill every week.
Is managed or self-hosted infrastructure better for enterprise AI agents? Managed platforms are faster to production and remove most runtime operations, which suits standard workloads. Self-hosting gives full control over data location, runtime and components, but your team owns scaling, isolation and on-call. Many enterprises choose a hybrid, with a managed runtime for the agent loop and their own data stores and tools.
How do you scale AI agents for thousands of users? Put a queue between user requests and agent workers, keep all state in external stores and autoscale workers on queue depth. Size the platform with Little’s Law, multiplying arrival rate by average task duration. Then check that model quota and downstream API limits can handle peak load, because extra workers cannot raise those limits.
How do AI agents resume work after a server failure? Agents resume when their progress is saved outside the process. The agent writes a checkpoint after each meaningful step, and the task stays on a durable queue until it completes. When a worker dies, the message becomes visible again, another worker loads the last checkpoint and continues. Idempotency keys stop completed actions from repeating.
What is the difference between an AI agent platform and AI agent infrastructure? An AI agent platform is a product that bundles tools for building, deploying and managing agents, often including a hosted runtime and connectors. AI agent infrastructure is the full set of underlying services an agent needs, whoever provides them. A managed platform covers part of that infrastructure, and your own data, tools and controls cover the rest.
