TL;DR
Context engineering picks what an AI model sees: one long document goes in whole, and large libraries go through retrieval. Numbers should come from database queries, since models slip on arithmetic. Bigger windows do not solve this, because models recall less as inputs grow. Four habits keep the window small: write, select, compress and isolate. Rank your sources, check who may see each one and log what the model received. Treat this layer as a security boundary, because a poisoned file can carry hidden instructions.
Key Takeaways
Context engineering covers everything in the model’s context window, from instructions and memory to retrieved files, tool definitions and tool results.
RAG, MCP and memory are parts of context engineering, while fine-tuning changes the model’s weights instead of what it sees.
Million-token windows still lose accuracy as they fill, and every extra token is billed on every call, so a smaller, high-signal context usually wins.
Write, select, compress and isolate are the four strategies that keep the window lean as agent tasks grow longer.
Permission checks, source ranking, memory expiry and logs of what reached the model belong in the pipeline, before the model ever sees the data.
Measure retrieval quality, groundedness and cost per successful task, then re-run a fixed failure set after every pipeline or model change.
An AI Agent With the Right Prompt and the Wrong Invoice
On a Thursday afternoon, an accounts payable agent receives an invoice from a packaging supplier. Its prompt is clear: check the price against the contract, then approve or flag. The agent then approves the invoice within seconds and writes a confident note explaining why.
Two weeks later, finance spots the problem. The supplier had signed a new contract with lower unit prices, but the search index still ranked the old PDF first. So the agent compared the invoice against a superseded document and found a perfect match.
Nobody had written a bad prompt. The model had simply been handed the wrong context.
The scenario is illustrative, yet the pattern appears wherever a company connects a model to real documents, records and tools. Rewording the instructions would not have fixed it either, because the prompt was never the problem. So this guide covers the discipline that does: the strategies, the memory rules, the pipeline, the costs and the governance behind a reliable context layer.
What Is Context Engineering?
Context engineering is the practice of deciding what enters a model’s context window on every call. That covers instructions and user input, as well as conversation history, memory, retrieved documents, tool definitions and tool results.
Anthropic published its guide to the discipline on September 29, 2025, once the term had caught on. It defines the goal as finding “the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome” on each call. The phrase rewards a second read, because it puts the right context ahead of more context.
For example, picture a capable new hire with no memory of your company. Before each task, someone decides which files land on that person’s desk, and the quality of the work depends on that pile. So context engineering is that decision, made by code on every call.
Most of what lands on the desk is unstructured data such as contracts, emails and tickets. That also makes context quality a data responsibility first. When a source is stale, mislabeled or open to the wrong people, no prompt can repair it.
Where the Term Came From
The phrase spread in June 2025, when two well-known voices backed it. Shopify CEO Tobi Lutke wrote that he preferred it to prompt engineering. He described it as “the art of providing all the context for the task to be plausibly solvable by the LLM.”
Andrej Karpathy replied with a “+1,” and Simon Willison’s write-up on June 27, 2025 carried the exchange to a wider audience. The fair credit, then, is that Lutke and Karpathy popularized the term together.
Karpathy’s own framing is the more practical one. He called it “the delicate art and science of filling the context window with just the right information for the next step.”
Those last three words carry the whole idea. Context is assembled one step at a time, which is why the Select strategy later in this guide matters so much.
What Sits in the Context Window
Every agent call assembles its window from several parts before the model sees anything. Each part has its own source, its own owner and its own way of going wrong. The table maps them for a typical enterprise agent.
Talk to Kanerika
Designing the Context Layer for an Agent?
Talk to Kanerika about source authority, token budgets, memory rules and ownership before your agent reaches production.
Talk to an Expert → Component Example in an enterprise agent Who owns it
System instructions Role, approval policy and output rules for the invoice agent AI engineering with the business process owner User input “Check invoice INV-2207 from Supplier A” The requesting user or workflow Conversation history Earlier turns in the same approval thread Application runtime Long-term memory Supplier preferences saved from past reviews AI engineering, with retention rules from compliance Retrieved documents The current supplier contract and pricing schedule Document owners in procurement Structured data and definitions PO lines, received quantities and the agreed meaning of “price variance” Data owners and data stewards Tool definitions Names and parameters for the ERP lookup and approval tools Platform engineering Tool results The purchase order record returned by the ERP lookup Source system owners Output schema JSON fields for decision, reason and cited sources AI engineering
Only one of these nine rows belongs to the person writing the prompt, even though the prompt gets most of the attention. That is the first sign of a team discipline, with data owners, platform engineers and security each holding a piece.
Context Engineering vs Prompt Engineering, RAG, MCP and Fine-Tuning
These five terms get used as if they compete, yet they sit at different levels of the same system. Prompt engineering shapes the instructions. Context engineering governs the whole window around them, and our comparison of how the two disciplines compare in production goes deeper.
RAG, by comparison, is one method inside that window. It fetches passages from a knowledge base when a query runs, and our guide to retrieval-augmented generation covers the architecture. Agentic RAG lets the agent decide when and how to search. Our breakdown of when retrieval beats a bare LLM then shows where retrieval earns its cost.
MCP is plumbing. Anthropic introduced the Model Context Protocol on November 25, 2024 as an open standard for connecting AI systems to data sources. As of November 2026, the specification stands at version 2026-07-28.
The protocol decides how tools and data reach the model, but never which ones should. Read what MCP is and how MCP differs from RAG before you treat it as a strategy in its own right.
Watch on YouTube
How MCP Makes AI Smarter, Faster, and Business-Ready
Kanerika explains how the Model Context Protocol connects AI models to enterprise tools and data, and what that changes for business use.
Approach What it changes When it changes Best for
Prompt engineering Instruction wording and examples When a template or release changes Clear task framing and output format Context engineering Everything in the window: instructions, data, memory, tools and state Every model call Agents and assistants on live enterprise data RAG Which documents get retrieved and inserted Whenever a query runs Large or changing knowledge bases MCP How tools and data sources connect to the model When integrations are added Standard, reusable tool access Fine-tuning Model weights Only when you retrain Stable style, format or domain behavior
Where Fine-Tuning Fits
Fine-tuning works on a different axis than context does. It changes the model’s weights through training, while context changes on every call. Our comparison of RAG and fine-tuning walks through that trade-off in detail.
The practical rule is short. If the model knows how to do the task but lacks the facts, fix the context. If it lacks the skill or the style, consider training, and expect prompt wording to matter in both cases.
Why 1M-Token Windows Still Do Not Solve the Problem
Frontier models now accept around a million tokens in one prompt, so a fair question is why selection still matters. If the window can hold a whole contract archive, then why not paste it all in? Three lines of research give the answer, so take them in turn.
The first is context rot. Chroma’s technical report of July 14, 2025 tested 18 LLMs from the mid-2025 generation. Performance grew increasingly unreliable as input length grew, and the decline was often uneven from one task to the next.
The second is position. The Lost in the Middle study by Liu and colleagues, posted to arXiv in July 2023, moved a relevant fact through long inputs. Models did best when the fact sat at the start or the end. Accuracy fell when the same fact sat in the middle.
The third is the attention budget. Anthropic’s guide notes that as the number of tokens in the window grows, the model’s ability to recall information from it decreases. So every token you add competes with the facts that matter.
Still, none of these studies tested the models on sale in November 2026. Treat them as a warning about a pattern, then test your own workload at the input lengths you actually plan to use.
Long windows also carry three operating costs that grow with every token:
Cost: every input token is billed again on every call the agent makes.Latency: longer inputs take longer to process before the first token comes back.Debugging: a wrong answer from a 200-page window is far harder to trace than one built from five documents.
What a Full Window Costs in November 2026
List prices make the trade-off concrete. The table shows published rates for six models, checked on November 2, 2026. Sources are the Anthropic pricing page , the OpenAI pricing page, Google’s model card and the xAI docs. Prices move often, so confirm them before you budget.
Model Context window Input / output per 1M tokens Cached input per 1M Long-prompt rule
Claude Opus 5.5 1M (128K output) $4 / $20 $0.20 Flat rate across the window Claude Sonnet 5.5 1M $2 / $10 $0.10 Flat rate across the window GPT-6.1 Sol 1,050,000 $2 / $10 $0.10 Prompts over 272K cost 2x input and 1.5x output GPT-6.1 Terra 1,050,000 $1 / $5 $0.05 Same 272K rule as Sol Gemini 4 Argon Up to 1M combined $2 / $10 through November 30, 2026; $4 / $20 from December 1, 2026 Not stated Not stated; price doubles December 1, 2026 Grok 4.7 500,000 $2 / $6 under 200K; $4 / $12 at 200K and over $0.50 under 200K; $1 at 200K+ Every token in a 200K+ request bills at the higher rate
Two worked sums show the gap, using list prices only. A 900,000-token prompt on GPT-6.1 Sol crosses the 272K threshold, so its input costs about $3.60 (0.9 x $2 x 2). The same prompt on Claude Opus 5.5 also costs $3.60, or about $0.18 when it is served from cache reads.
Now compare a budgeted window. A 12,000-token prompt on Sol costs about $0.024 in input, roughly 150 times less per call than the full window. At an assumed 10,000 calls a day, that is about $36,000 against $240 in daily input spend.
The rule is simple. Window size is a ceiling and a bill, never a strategy, and the rest of this guide is about filling that space well.
The Four Core Strategies: Write, Select, Compress, Isolate
Once the window is treated as scarce, the next question is how to manage it. LangChain’s breakdown of July 2, 2025 groups the techniques into four strategies.
Strategy What it does Common technique Enterprise example
Write Saves information outside the window for later Scratchpads and long-term memory stores An agent records which invoices it has already flagged Select Pulls only relevant information into the window Retrieval, just-in-time loading and tool selection Loading the current contract instead of every version Compress Shrinks what is already in the window Summarization, compaction, trimmed tool output and prompt caching Summarizing a 40-turn thread while keeping invoice IDs Isolate Splits context across separate windows Sub-agents and state objects outside the prompt A research sub-agent reads 30 contracts and returns a short brief
Write: Scratchpads and Memory
Writing context means saving it outside the window so the agent can reach it later. In LangGraph, for example, short-term memory is checkpointed per thread, which gives the agent a scratchpad across the steps of one task. Long-term memory persists across sessions and holds facts such as a supplier’s preferred invoice format.
The hard part, however, is deciding what to keep. A fact written without a source or an expiry date eventually turns into a liability, which the memory section below covers in full.
Select: Retrieval, Just-in-Time Loading and Tool Choice
Selecting context means pulling in only what the current step needs, when it needs it. Retrieval is the familiar form. Anthropic also describes a “just in time” approach. Here the agent keeps lightweight identifiers, such as file paths or record IDs, and loads the data only when a step calls for it.
Tool choice is selection too, because every tool definition costs tokens on every call. An agent with dozens of tools reads all of their definitions each time, so expose only the tools a task needs.
Relationship-heavy data, though, needs one more option. A knowledge graph approach to RAG can select connected facts, such as which supplier ships which part to which plant, that plain vector search misses. A graph helps only while it is current, because a stale graph returns confident links that no longer exist.
Checklist
Agentic AI Readiness Checklist
Check whether your data, controls and monitoring are ready before AI agents run on enterprise context.
Get the Checklist → Compress: Summaries, Compaction and Caching
Compressing context means shrinking what is already in the window so the next step has room. Anthropic calls one form of this compaction, in which a long conversation is summarized and the agent then continues from the summary. Trimming verbose tool output is also a quick win.
The risk, however, is losing the detail that mattered. A summary saying “three invoices were disputed” is useless when the next step needs their IDs. So keep identifiers, amounts, exceptions and open decisions verbatim, and summarize only the narrative around them.
Prompt caching, meanwhile, is the cost side of compression. Put the stable parts of the window, such as instructions, tool definitions and policy text, at the front so the provider can cache that prefix. As of November 2026, a cache read on Claude Opus 5.5 costs $0.20 per million tokens, against $4 for fresh input.
Isolate: Sub-Agents and State Outside the Prompt
Isolating context means splitting work so no single window has to hold everything. For instance, a coordinating agent can hand a research task to a sub-agent with a clean window. The sub-agent reads widely, then returns a short, structured result.
Similarly, state objects keep detail out of the window. Keep the full order record in application state, then show the model only the fields the current step needs. The pattern runs through agentic AI workflows that chain several steps, and our guide to orchestrating sub-agents covers how coordinators route that work.
Managing Context Memory Across Sessions and Agents
In context engineering, memory is the part of context that outlives a single call, which makes it both useful and risky. A fact the agent stores on a Monday can shape a decision months later, long after anyone remembers where it came from.
In practice, three scopes cover most enterprise designs. Thread memory lives for one task or conversation, while user memory holds one person’s preferences across sessions. Shared agent memory holds facts that a whole fleet of agents can read, so it carries the widest blast radius.
Each scope needs written rules before the first fact is stored:
Validate: check a fact against its system of record before writing it.Source: store where the fact came from, with the document version or record ID.Owner: name the team that can correct or delete the entry.Expiry: give every entry an end date, such as 90 days for supplier preferences.Scope: decide which users and agents may read it, under the same access rules as the source.
Without those rules, memory drifts. A one-off exception, such as a waived late fee, becomes the default for every later invoice. Nobody can then trace why the agent behaves that way.
Memory is also an attack surface. An injected instruction that gets written to long-term memory can resurface in later sessions, after the poisoned document is gone. Memory writes therefore need the same checks as retrieval.
Handoffs between agents also follow one rule. Pass a structured summary with IDs, decisions and open questions, never the full transcript, so the receiving agent starts with a lean window.
Some teams go a step further and let agents update their own instructions from experience. Our guide to agents that refine their own playbooks covers that pattern and the review gates it needs.
How to Build a Context Engineering Pipeline Step by Step
Strategies describe what to do, while a pipeline makes them repeatable. Every call is assembled the same way, so it can be audited later. The seven steps below work for one assistant as well as for a fleet of agents.
Define the task and what success looks like. Write down the decision the model must make and how you will know it was right. “Approve invoices that match the active contract within tolerance” is testable, while “help with invoices” is not.Map sources and rank their authority. List every source the task might use and decide which one wins in a conflict. A signed contract should outrank an email thread that quotes it.Choose the retrieval method per source. Use semantic search for contracts and policies, SQL or API calls for transactional records, and direct lookups when you already know the ID.Filter, rank and permission-check. Drop superseded versions, rank by relevance and authority, and enforce the user’s access rights before anything reaches the window.Allocate the token budget. Reserve space for instructions, the request and the output first. Then split the rest among sources by priority.Manage memory deliberately. Decide what gets written, who can read it and when it expires, using the rules in the previous section.Validate and log. Check that tool results are fresh and consistent, then record which sources reached the model on every call.
Step 4 is where the invoice agent failed, because nothing filtered out the superseded contract. A version filter or an authority rank would have kept it out of the window.
A Reference Architecture From Source Systems to Logged Context
The seven steps map onto a reference architecture that most enterprise stacks can host. Data flows from source systems through ingestion, where each document picks up metadata such as version, owner, status and effective date.
Once ingested, it lands in two kinds of store. Text goes to a search index, while records stay in the warehouse and are reached through SQL or an API at query time.
The next layer filters, ranks and checks permissions, and it sits before the token budget for a reason. Anything that passes this layer competes for space, so an unchecked stale file can push out the current one.
After assembly, the model answers, and then a validation step checks the output and any tool results. A trace log then records every source ID, version and token count. That log turns a wrong answer into a short investigation instead of a two-week mystery.
Writing the Context Budget Down
A context manifest makes steps 2 to 6 explicit, so a reviewer can check each choice. Treat the YAML below as a template, since its values are illustrative. The reserved space and source caps add up to 8,600 tokens, which leaves headroom under the 12,000-token ceiling.
# Template: values are illustrative, not recommendations
task: invoice_price_check
budget_tokens: 12000 # ceiling for one call
reserve:
system_instructions: 1500
user_request: 300
output: 800
sources: # highest authority first
- name: active_contract
method: semantic_search
filter: {status: active, supplier_id: "$supplier"}
max_tokens: 4000
- name: purchase_order
method: sql
query: po_lines_by_invoice
max_tokens: 1500
- name: supplier_memory
method: memory_lookup
max_age_days: 90
max_tokens: 500
log: [source_ids, versions, token_counts]Treat the manifest like code. Version it, review every change and run it against known failure cases before it reaches production.
Who Owns the Context Layer
No single team owns context end to end, so gaps appear between them. Data owners know which sources are authoritative, while engineers know how retrieval works. Security also knows who may see what.
Role Owns Example decision
Data owners and stewards Source authority, definitions and freshness Which contract repository is the system of record Platform and AI engineering Retrieval, memory, token budgets and logging How many tokens the contract source may use Security and compliance Access policies, PII rules and retention Whether memory may store supplier bank details Business SMEs Success criteria and test cases What counts as an acceptable price variance
Put all four groups in one design review before an agent ships. One shared review catches several of the mistakes described later in this guide before they reach production.
Choosing a Context Approach for Your Use Case
No single architecture fits every task, because enterprise data comes in very different shapes. The right approach depends on that shape, how often it changes and where it already lives.
A useful habit is to start from the data and only then work back to the method. A finance question lives in a warehouse, a policy question lives in documents, and an order status lives in a live system. So each one needs a different route into the window.
Use case Data shape Approach that fits Where it lives
Summarizing one long report One bounded document Long context, since the whole document fits A single file Policy and contract Q&A Large, changing document set RAG with version and permission filters Vector index plus metadata Finance or sales metric questions Tables and transactional records Structured retrieval through SQL or a semantic layer Warehouse or semantic layer Multi-step operational agents Live systems and APIs Just-in-time tool calls with state outside the prompt Live APIs Personalized assistants User preferences over time Long-term memory with expiry and consent rules Memory store Research across many sources Large, mixed corpus Sub-agents that return compressed findings Sub-agent outputs
Numbers deserve a special warning. Pasting a spreadsheet into the window invites the model to do arithmetic it may get wrong, so query the figure and pass the result instead. If those figures already flow through a governed data analytics pipeline , point the agent at that rather than at raw exports.
Document-heavy work such as contracts and policies is where retrieval quality decides the outcome. For that reason, version filters and permission checks belong inside the retriever itself. That way, stale or restricted files never reach the ranking step at all.
Kanerika Service
RAG Development Services
Kanerika designs retrieval pipelines with version filters, authority ranking and permission checks, so agents answer from current, approved sources.
Explore RAG Development Worked Example: The Invoice Agent, Rebuilt
Here is how the agent from the opening assembles its window after a redesign. Each step adds only what the next decision needs, so the window stays small.
The system instructions load the role, the approval policy and the rule that the active contract outranks every other source.
The request arrives with an invoice ID, so the agent calls the ERP tool and receives the purchase order lines.
Retrieval searches contracts filtered to status “active” for this supplier and returns only the current pricing schedule.
A memory lookup adds one validated note: this supplier usually bills freight separately.
The agent compares prices, finds one line above the allowed variance and checks the tolerance in the policy.
It flags the invoice and writes a reason that cites the contract version and the PO number.
The pipeline logs every source ID, version and token count for the audit trail.
The prompt barely changed between the two versions, while the window changed completely. What changed was what reached the model, and in what order.
Seven Context Mistakes That Break Enterprise AI
The same handful of context errors shows up once agents leave the demo stage. Each one is easy to make, and each is cheap to prevent when caught early.
Treating it as prompt writing. Teams polish the instructions and ignore the sources behind them. The invoice agent had a fine prompt and a stale contract.Assuming RAG solves everything. Retrieval ranks text by similarity, so the top result is not always current or authoritative.Stuffing the window. Adding every possibly relevant document feels safe. It dilutes attention and buries key facts in the middle.Giving every source equal trust. A signed contract, a wiki page and a forwarded email are not equally reliable. Encode the ranking so the model does not have to guess.Ignoring permissions at retrieval. Once the retriever can see a document, the model can quote it. Enforce access rights before content enters the window.Treating memory as permanent storage. Memory without expiry turns an old exception into a standing rule. Give each entry a source, an owner and an end date.Evaluating only the final answer. A correct answer built on the wrong source is luck. Score retrieval and groundedness separately to see which layer failed.
Sources hold mistakes 2 and 4, the window holds 1 and 3, access and memory hold 5 and 6, and evaluation holds 7. To find the weakest layer in your own stack, then, start with an honest assessment of your data and controls.
AI Assessment
Is Your Data Ready for Enterprise AI Agents?
Kanerika’s AI Maturity Assessment scores your data, governance and skills, so you can see where context gaps sit before agents scale.
Start Your AI Assessment → Governance and Security at the Context Layer
Everything that enters the context window can change what the model does. That makes the context layer a security boundary as well as a quality concern.
OWASP lists prompt injection as LLM01:2025 , the first entry in its Top 10 for LLM applications. The entry covers indirect injection, where a model takes input from external sources such as websites or files. It also notes that RAG and fine-tuning do not fully mitigate the risk.
In an enterprise agent, for example, a poisoned vendor PDF or a crafted email is enough. Once retrieval pulls it in, the model reads the hidden instruction as context. Nothing in the prompt can then tell it apart from real data.
Five Controls That Sit Before the Model
Memory widens that surface, since a poisoned instruction written to memory can return in later sessions. So the controls that work all sit before the model, where they can stop bad content from entering the window:
Permission-aware retrieval: filter documents by the requesting user’s rights at query time.Source allow-lists: let agents retrieve only from approved repositories and domains.Lineage of context: log which document, version and chunk reached the model on each call, so audits can replay it.PII filtering: mask personal and secret fields before they enter the window or memory.Action limits: require human approval for high-impact actions such as payments.
These controls also build on data governance work many enterprises already run. Microsoft Purview sensitivity labels can feed the filters, and Azure AI Search can ingest those labels and enforce document-level access at query time. As of November 2026 that feature is in preview, with no SLA, and Microsoft does not recommend it for production.
When you need a risk framework to hang these controls on, use NIST AI 600-1 . It is the Generative AI Profile of the NIST AI Risk Management Framework, published on July 26, 2024.
For the wider threat model, see our guide to LLM security risks and controls . Our guides to Microsoft Purview data loss prevention and AI guardrails cover the policy layer and the runtime checks around it.
How to Measure Whether Your Context Layer Works
A good final answer can hide a weak context layer, especially when nobody checks the sources. Measurement therefore has to look at what was retrieved, how it was used and what it cost.
Metric What it tells you How to collect it
Retrieval precision Share of retrieved chunks that were relevant Label a sample of retrievals against a test set Retrieval recall Share of needed facts that were retrieved Gold-answer test set with known source documents Groundedness Whether the answer is supported by the supplied context LLM-as-judge or human review on sampled traces Context tokens per call How lean the window is Token counts from logs, tracked per task type Latency Time to first token and to the full response Application traces Cost per successful task Spend divided by tasks that passed validation Billing data joined with outcome checks Failure-test pass rate Resilience to injection, stale and conflicting sources Scheduled red-team test suite
Groundedness and correctness are different checks, even though teams often treat them as one. The invoice agent’s first answer was fully grounded, since it matched the contract it was given. It was still wrong, because that contract had been superseded.
Cost per successful task also deserves attention. Cost per call can fall while cost per success rises, because cheaper calls may fail more often and need retries.
Drift Checks and Judge Models
Context quality also drifts as sources, pipelines and models change. Keep a fixed regression set with a superseded contract, two sources that disagree and a document with hidden instructions. Re-run it after every pipeline or model change, just as you would run unit tests.
When an LLM judge scores groundedness, pick a judge from a different model family than the one under test. That matters because two models from one family tend to share blind spots.
Our LLM evaluation framework guide covers judge design in depth. The guide to AI agent observability covers the tracing that feeds these metrics.
Context Engineering Tools and Frameworks
Tools make the work easier, but none of them does it for you. Think in roles first, then pick a product that fills each role and accept its trade-off.
Role What it does Example product Trade-off
Orchestration and memory Manages agent state, thread checkpoints and long-term memory LangGraph Fine-grained control, with more code for your team to own Agent framework Builds and runs single agents and multi-agent workflows Microsoft Agent Framework (Python and .NET; Go in public preview) Natural fit on Microsoft stacks; Go support is still in preview Tool and data connection Standard interface between models and tools or data Model Context Protocol (MCP) Reusable connectors, yet every new server is a new trust boundary Retrieval store Stores embeddings and runs similarity search Chroma Quick to start; version filters and permissions are yours to design Catalog and definitions Holds business definitions, ownership, lineage and labels Microsoft Purview Strong governance record, but agents need a retriever that reads it Observability and evaluation Traces calls, scores groundedness and runs test sets LangSmith Tightest fit with LangChain and LangGraph stacks
Notes on the Microsoft Stack and the Retrieval Layer
Microsoft Agent Framework needs a note on its history. Microsoft describes it as the direct successor to Semantic Kernel and AutoGen , built by the same teams. So new agent work on the Microsoft stack should evaluate it first.
The retrieval and catalog rows are still the ones teams underfund. Our vector database comparison and our review of data catalog tools compare the options. For the rest of that stack, see our roundup of RAG tools for the retrieval layer .
Kanerika Service
LLM Development Services
Kanerika selects, integrates and evaluates language models for enterprise workloads, with context, retrieval and governance designed in from the start.
Explore LLM Development Model choice matters less than this plumbing. Swapping the model would not have fixed the invoice agent, because every model would have read the same superseded contract.
A hosted frontier model and a small private one fail the same way when the window is wrong. Spend the first engineering hours on sources, filters and logs, then compare models on a workload that already has clean context.
How Kanerika Builds Context Layers for Enterprise AI
Kanerika treats context design as a data problem first and a model problem second. As the invoice agent showed, failures usually start in the sources an agent reads, so that is where the work begins. The method rests on data-platform depth, and Kanerika is a Microsoft Solutions Partner for Data and AI with the Analytics Specialization.
An investment bank shows the pattern in practice. Its employees searched document repositories and queried databases through separate interfaces, and access controls sat outside retrieval. Every answer therefore meant two searches and a manual check of who was allowed to see what.
Kanerika then deployed KlarityIQ for unstructured RFI documents and Karl for structured databases, behind one chat interface. Role-based access control was also applied at the retrieval layer itself. The engagement delivered three measured results:
43% faster information retrieval
100% role-based compliance at the point of retrieval
35% higher workforce efficiency
Case Study
43% Faster Information Retrieval for an Investment Bank
KlarityIQ and Karl gave employees one chat interface across documents and databases, with role-based access enforced at the point of retrieval.
Read the Case Study → Read through this guide’s lens, documents went through retrieval and numbers went through queries, as the use-case table recommends. Permissions were enforced before anything reached the window, which is the first governance control above.
The Four-Stage Method
Most engagements follow four stages:
Audit failing workflows. We trace each wrong answer back to the context behind it.Ready the data layer. That means a catalog of sources, agreed definitions and access policies the retriever enforces, built on our data governance services .Design retrieval, memory and tool context with governance built in. Permission filters, source ranking and lineage logs are part of the design from day one.Test with a context failure set, then monitor. Stale documents, conflicting sources and injection attempts run before launch and after every change.
Three Kanerika products also cover common context jobs. KlarityIQ handles document retrieval, and Karl gives governed answers from structured data. Susan redacts PII before content enters the window or memory.
Our advice for a first project rarely changes, since the same failures keep appearing. Start with one agent, log two weeks of traces before changing anything and keep the failure set in version control. Every later fix is then tested against the last one.
A 30-60-90 Day Plan for Your First Context Layer
You do not need a platform rebuild to begin context engineering. Instead, pick one agent that already touches business data and give it 90 days, with an exit test at the end of each stage.
Days 1 to 30: See What the Model Sees
Log what reaches the model on every call, because you cannot fix a window you cannot see. Then list every source the task uses and rank each one by authority, with the system of record on top. The exit test is a trace for every call and a ranked source list that the data owner has signed.
Days 31 to 60: Filter, Budget and Test
Add version filters and permission-aware retrieval, so superseded and restricted files never reach the window. Also write the token budget manifest and keep it in version control next to the code. Then build a 10-case failure set covering stale documents, conflicting sources and at least one injection attempt.
Pull those cases from real traces wherever you can. The superseded contract from the opening makes a strong first case, because it fails silently and the answer still looks correct. A case that fails loudly teaches the team far less than one that fails quietly.
Track context tokens per call once this stage starts. If the number climbs while accuracy stays flat, a source cap is too loose, and the manifest is the place to tighten it. Keep the failure set small enough to run in minutes, so nobody is tempted to skip it.
The exit test for this stage is simple. The failure set passes, and every change to the pipeline runs it first.
Days 61 to 90: Memory, Drift and Sign-Off
Write the memory rules for validation, source, owner, expiry and scope before the agent stores anything long term. Also schedule drift checks that re-run the failure set after every pipeline or model change. Then take the design to security, compliance and the business owner for a governance sign-off.
Start memory narrow. At first, let the agent write only to thread memory. Then open user or shared memory once the rules have run for a few weeks without a bad entry.
Treat a model upgrade like any other pipeline change, because it is one. A new model can read the same window differently, so the failure set runs before the switch. Give the drift checks a named owner as well as a schedule, or the first failing run gets ignored.
The sign-off review should then answer four questions in writing. Who can see each source? What may the agent store, and for how long? Which actions need a human, and where does the trace log live?
Expand to the next agent only after the first one proves its context quality, access control and task accuracy. Reuse the manifest, the failure set and the trace log as templates, so each new agent starts from a tested baseline.
For an enterprise, the payoff is control more than any single model. A context layer with ranked sources, permission checks and trace logs lets you swap models, add agents and pass audits without starting over.
The invoice agent from the opening never needed a better prompt. It needed a version filter, a token budget and a trace log. That is what context engineering for enterprise AI comes down to, so the right data reaches the model under rules you can audit.
Kanerika Service
Review Your Context Layer
Kanerika’s AI governance team reviews permissions, source controls, memory rules and logging, so your agents meet policy before they scale.
Explore AI Governance Frequently Asked Questions
What is context engineering in AI?
Context engineering is the practice of deciding what enters an AI model’s context window on every call. That covers instructions, user input, history, memory, retrieved documents, tool definitions and tool results. Models can only reason over what they receive. A clean, current context produces grounded answers, and a stale or noisy one produces confident mistakes.
What is a context engineer?
A context engineer designs the systems that assemble what an AI model sees. The role covers retrieval, memory, tool access, token budgets and logging, working closely with data owners and security. In many enterprises it is a responsibility within AI or platform engineering rather than a separate job title. The skills sit between data and AI engineering.
What is context rot?
Context rot is the decline in model performance as the input grows longer. Chroma’s technical report of July 14, 2025 tested 18 LLMs and found performance grew increasingly unreliable with input length, often unevenly. Anthropic describes the same effect, since recall falls as tokens in the window increase. Test your own workload at the lengths you plan to use.
Does a larger context window remove the need for context engineering?
No. A larger window raises the ceiling but does not decide what belongs inside it. The Lost in the Middle study found models often miss facts placed in the middle of long inputs. Every extra token also adds cost and latency on each call. A lean, well-ordered context usually beats a full window of loosely relevant material.
How do you measure context quality?
Measure the context separately from the final answer. Track retrieval precision and recall against a labeled test set, groundedness in the supplied sources, context tokens per call, latency and cost per successful task. Keep a failure set with stale documents, conflicting sources and injection attempts. Run it after every pipeline or model change, like a unit test suite.
How can enterprises prevent prompt injection in retrieved data?
Treat retrieved content as untrusted input. OWASP lists prompt injection as LLM01:2025 and notes that RAG and fine-tuning do not fully mitigate it. Useful controls include permission-aware retrieval, allow-lists of approved sources, logs of which documents reached the model, masking of sensitive fields and human approval before an agent makes a payment or similar action.
What are the four pillars of context engineering?
The four pillars are write, select, compress and isolate, a grouping LangChain described on July 2, 2025. Write saves information outside the window for later steps. Select pulls in only what the current step needs. Compress summarizes history while keeping IDs and exceptions. Isolate splits work across sub-agents or state objects so no single window holds everything.
Is context engineering replacing prompt engineering?
No. Prompt engineering becomes one part of context engineering rather than disappearing. Instructions still need careful wording, but in agents and enterprise assistants the instructions are a small share of what reaches the model. Retrieved documents, memory and tool results now carry most of the risk, so the discipline that manages all of them has grown in importance.
How do you keep AI context from drifting over time?
Drift comes from changing sources, pipelines, models and memory, so each needs a check. Tag documents with versions and effective dates, give every memory entry a source and an expiry date, and log what reached the model. Then re-run a fixed regression set after every pipeline or model change and compare the scores with the last run.
What makes enterprise data ready for AI agents?
Data is ready when an agent can find the authoritative version, understand it and only see what the user may see. That means a catalog of sources with owners, agreed business definitions, version and status metadata on documents, and access policies the retriever enforces. Numbers should come from governed queries, never from pasted spreadsheets in the window.
What is the difference between context engineering and prompt engineering?
Prompt engineering shapes the wording and examples in the instructions. Context engineering governs everything in the window, including instructions, retrieved data, memory, tool definitions, tool results and output format. A prompt is usually written once per template, while context is assembled fresh on every call. Agent failures often trace back to what reached the model, not the phrasing.
What are the components of context engineering?
The main components are system instructions, user input, conversation history, long-term memory, retrieved documents, structured data and business definitions, tool definitions, tool results and an output schema. Each has its own owner and its own failure mode. Context engineering decides which of them enter the window for a given step, in what order and within what token budget.
What tools or frameworks support context engineering?
Useful tools fall into roles. LangGraph manages agent state and short-term and long-term memory. Microsoft Agent Framework, the successor to Semantic Kernel and AutoGen, builds and runs agents. MCP standardizes tool and data connections, and vector databases such as Chroma handle retrieval. Data catalogs and observability tools complete the stack by supplying definitions and traces.
Does context engineering require custom tools or platforms?
Not usually. Most teams start with an existing agent framework, a vector store and the data platforms they already run. The custom work lies in the rules, such as which sources count as authoritative, how retrieval filters by version and permission, what memory keeps and how each call is logged. Those rules are specific to your business.
Is context engineering relevant for small teams or only large enterprises?
It matters for any team that connects a model to real data. A small team with one support assistant still needs current documents, sensible memory and a way to see what the model received. The scale changes the tooling and the number of owners involved, not the principles. Starting small actually makes it easier to build good habits early.
Who coined the term context engineering?
No single paper coined it. The term spread in June 2025, when Shopify CEO Tobi Lutke wrote that he preferred it to prompt engineering and Andrej Karpathy replied with a +1. Simon Willison documented the exchange on June 27, 2025. Lutke and Karpathy are best described as the two people who popularized the phrase together.
Is context engineering the same as RAG?
No. RAG, or retrieval-augmented generation, is one method inside context engineering. It fetches relevant passages from a knowledge base and inserts them into the window at query time. Context engineering also covers memory, tool selection, compression, isolation, token budgets, permissions and logging. A system can use RAG well and still fail because other context is stale.
How is context engineering different from fine-tuning?
Fine-tuning changes a model’s weights through extra training, so the change persists across every call. Context engineering changes what the model sees on each call, with no retraining. Use context when the model has the skill but lacks current facts. Consider fine-tuning when it needs a stable style, format or domain behavior that instructions cannot produce.
