TL;DR
The best AI observability tools in 2026 depend on your stack: pick Langfuse or Comet Opik if you must self-host, LangSmith if you build on LangChain, and Datadog if you already run it. Regulated teams that run classic ML next to LLMs should shortlist Arize or Fiddler, and Azure teams can start with Microsoft Foundry observability. Every tool here traces prompts, retrieval, tool calls and responses, then scores the answers, because a 200 OK reply can still be wrong. As of November 3, 2026, list prices start free, with Langfuse Core at $29 a month and LangSmith Plus at $39 per seat. Ownership keeps shifting, since ClickHouse acquired Langfuse and CoreWeave acquired Weights & Biases, so insist on OpenTelemetry export and test that a trace survives the move. Roll out in four stages: instrument, define failure modes, add evaluation, then connect the evidence to governance.
Key Takeaways
AI observability tools record what a model saw, did and said, so teams can see why an answer was wrong even when every system metric looks healthy.
Pick by stack first: Langfuse or Opik for self-hosting, LangSmith for LangChain, Datadog for Datadog shops and Foundry observability on Azure.
Arize and Fiddler suit regulated enterprises that monitor classic ML models and LLM applications side by side.
List prices run from free tiers to $249 a month for Braintrust Pro and $2,499 a month for Langfuse Enterprise, as of November 3, 2026.
Accepting OpenTelemetry data is not enough: check that span links, model ids, token counts, tool calls, eval scores and session ids survive a move to a second backend.
Judge the system by cost per successful task, and keep the traces as audit evidence for the EU AI Act’s logging rule and the NIST AI Risk Management Framework.
A Refund Answer That Passed Every Check
Picture a customer-support agent built on a large language model (an illustrative example). A customer asks how refunds work, and the agent replies in two seconds with a polite, confident answer. The answer quotes a refund policy the company retired three months ago.
Nobody on the engineering team notices, because nothing technically failed. The complaint reaches the support lead a week later, and by then other customers may have read the same retired policy.
The Agent Returned 200 OK and Still Gave the Wrong Answer
Every dashboard stayed green during that failure. The API returned 200 OK, latency sat under target and no error fired, so classic monitoring reported a healthy system.
That gap is what AI observability tools close. Monitoring, after all, asks only whether software ran. An AI system also needs an answer to a harder question, which is whether the output was right, grounded and safe.
Agents make the question harder with every step they take. One request can include retrieval, several model calls and a chain of tool calls, and a wrong answer can start at any of them. This shift is part of how AI agents have evolved from single prompts into multi-step workflows.
The ten tools below are the ones we would shortlist for an enterprise as of November 2026. Each review gives the best fit, one watch-out and a checked list price. After the reviews come a selection method, a cost model and a four-stage rollout plan.
What AI Observability Tools Do (and Where APM Stops)
AI fails quietly, so an observability tool for AI has to capture meaning as well as uptime. Application performance monitoring (APM) tracks requests, latency and errors across services. That view stays useful, but it cannot tell you whether an answer was correct.
AI observability adds a record of what the model actually worked with. For each request it stores the prompt, the retrieved context, every tool call, the response and a quality score. In the refund case, the trace would show that retrieval pulled an outdated policy document. A groundedness check would then flag the answer before more customers saw it.
The two layers work together rather than compete. APM tells you the system is up, while AI observability tells you the system is right. If you are adding the instrumentation yourself, our guide on how to instrument LLM apps covers the code-level setup.
AI Observability vs LLM Monitoring vs APM vs AIOps vs Data Observability
These five terms overlap in vendor marketing, so shortlists get confusing. The table separates them by what each one watches and what it tells you.
Discipline What it watches What it tells you Example tools
APM Requests, latency and errors across services Alerts when requests fail or slow down Datadog APM, New Relic, Dynatrace LLM monitoring Model calls, tokens, cost and latency Alerts when model calls exceed budget or speed targets Usage dashboards in most tools below AI observability Full traces of prompts, context, tool calls and outputs, plus evaluation scores Why an answer was wrong, even when the request succeeded Arize, Fiddler, Galileo, Braintrust AIOps IT alerts, events and incidents Groups related alerts so on-call teams act faster See our guide to AIOps tools Data observability Pipelines, freshness, schema and volume Alerts when data feeding AI arrives late or broken See our list of data observability tools
A production AI system usually needs at least three of these layers. Bad data upstream, a slow service and a wrong answer can all arrive as the same customer complaint. Only the right layer, in practice, tells them apart.
Five Signals Worth Capturing
A useful AI trace carries five kinds of signal. Each one answers a different question, and each one needs a named person who acts on it.
Traces and spans. One trace per request, with a span for retrieval, each tool call and each model call, so you can see which step went wrong. Platform and ML engineers own this signal.Quality scores. Groundedness, relevance and policy-match scores sit on the trace, next to user feedback such as thumbs-down votes. ML engineers and QA act on them.Tokens and cost. Tokens per request, rolled up into cost per resolved conversation. FinOps and product owners watch this number.Latency. Time per span as well as per request, because a slow retriever and a slow model need different fixes. Platform engineers own the alert.Safety. Prompt injection attempts, personal data in prompts or responses, and policy violations. Security and risk teams review these events.
Kanerika Service
Agentic AI Development
Kanerika designs and deploys governed AI agents with tracing, evaluation and approval gates built in from the first release.
Explore Agentic AI Services Agents add a twist to the first signal. Every tool call is its own span, including calls an agent makes through the Model Context Protocol (MCP) to outside tools and data. If those spans are missing, the trace shows a model call that produced a bad answer but hides the tool result that caused it.
The trace is also a record of what the model saw. That context is something your team designed, so capturing it lets you debug your context engineering instead of guessing at it.
Capture all five signals from day one. A team that logs only tokens and latency will catch cost spikes but miss the wrong answer every time.
How We Picked These 10 Tools
A tool earned a place here only if an enterprise team could run it in production, not just in a notebook. We scored each candidate against seven criteria, then checked vendor pages and pricing on November 3, 2026.
Agent tracing. Can it follow one request across retrieval, tools and several model calls? Our guide to AI agent observability covers what to trace and alert on for agents.Evaluation depth. Does it score outputs offline and in production, with both automated and human review?OpenTelemetry support. Can traces flow in and out using an open standard, with their fields intact?Deployment choice. Is there a managed service, a private-cloud option or full self-hosting?Enterprise controls. Look for single sign-on, role-based access, retention settings and audit logs.Pricing clarity. A public list price or a clear free tier scored higher than quote-only plans.Vendor stability. We checked ownership and license status at primary sources such as vendor announcements and repositories.
Narrower prompt-management tools did not make the cut, and neither did usage dashboards that log tokens without scoring answers. We also cover ownership changes in a separate section, because a buyer needs that context before signing a multi-year contract.
One caveat applies to every review below. These rankings rest on vendor documentation, public pricing and license files rather than a benchmark we ran ourselves. A pilot on your own traffic is the test that settles the choice, and the Kanerika section describes how to run one.
Talk to Kanerika
Narrowing Your AI Observability Shortlist?
Kanerika reviews your AI stack, data sensitivity and team size, then recommends the two or three observability tools worth a pilot.
Talk to an Expert → AI Observability Tools Compared at a Glance
Here is the whole shortlist of AI observability tools on one screen, before the detailed reviews. Prices are public list prices checked on November 3, 2026, so confirm them before you budget.
Tool Category Best for Open source or self-host Entry price (list, Nov 3, 2026)
Langfuse Open-source LLM engineering platform Teams that want to self-host Yes, open-source core Free Hobby tier; Core $29/month Arize (Phoenix, AX) ML and LLM observability Mixed ML and LLM estates Phoenix under Elastic License 2.0 AX Free $0; AX Pro $50/month LangSmith LLM tracing and evaluation LangChain and LangGraph teams Managed service Developer $0; Plus $39/seat/month Datadog Agent Observability AI layer inside an APM suite Teams that already run Datadog Managed service $160/month for the first 100,000 LLM spans, billed annually Braintrust Evaluation-first platform Release gating on eval scores Enterprise on-prem or hosted Starter $0; Pro $249/month Comet Opik Open-source LLM observability Free full self-hosting Yes, Apache-2.0 Free to self-host; Pro Cloud $19/month Fiddler AI Agent control plane plus ML monitoring Regulated enterprises Ask about private deployment Pricing not public Galileo Evaluations that become guardrails Hallucination and safety checks at scale Ask about private deployment On vendor pricing W&B Weave LLM tracing inside Weights & Biases Teams already on W&B Ask CoreWeave about deployment On vendor pricing Microsoft Foundry observability Azure-native tracing and evaluation AI built on Azure Azure service Billed through Azure
Reading the Table
Two columns need care before you compare. Usage units differ by vendor, so a “trace” in one tool is not a “unit” in another. Compare quotes on your own monthly volume instead of headline prices.
Self-hosting removes the license fee but not the cost. Someone still runs the database, applies upgrades and answers the pager. Because that saving is smaller than it looks, the stack you already run usually decides the vendor.
The 10 Best AI Observability Tools in 2026
Each review covers what the tool is, where it fits best, one watch-out and its price line. The order follows the shortlist logic above, starting with the open-source options.
1. Langfuse: Best Open-Source Option for Self-Hosting
Langfuse is an open-source LLM engineering platform. It combines tracing, prompt management, evaluation scores and datasets in one product. Teams can also run it on their own infrastructure instead of the managed cloud.
Ownership changed on January 16, 2026, when ClickHouse acquired Langfuse . The announcement also says Langfuse remains open source, with its core features kept under the existing MIT license.
Langfuse especially fits teams whose traces hold customer conversations that must stay inside the network. The watch-out is operational, because self-hosting moves upgrades, storage growth and uptime onto your own team.
The Langfuse pricing page lists a free Hobby tier with 50,000 units a month and two users. Core costs $29 a month, Pro $199 and Enterprise $2,499. Each paid plan includes 100,000 units, with $8 per extra 100,000 units and lower rates at volume.
2. Arize AI (Phoenix and Arize AX): Best for Mixed ML and LLM Estates
Arize comes in two parts. Phoenix is the source-available project, with tracing built on OpenTelemetry and released under the Elastic License 2.0, according to its GitHub repository . Arize AX, in contrast, is the managed platform for production monitoring.
Arize started in classic ML monitoring, so it watches drift and model performance alongside LLM traces. That suits enterprises with fraud or forecasting models already in production, since one team can watch both kinds of model with the same MLOps practices .
The watch-out is the license. Elastic License 2.0 is source-available rather than OSI open source, so legal teams should review it before Phoenix ships inside a product. The Arize pricing page lists AX Free at $0, Pro at $50 a month and a custom Enterprise plan.
3. LangSmith: Best for LangChain and LangGraph Teams
LangSmith comes from the team behind LangChain, so the two fit closely. Agents built on LangChain or LangGraph get tracing with very little setup, and each step of a graph shows up as its own span.
That native fit pays off in AI agent orchestration , where several agents pass work between them. You can then replay a failed run, compare prompts and turn bad traces into test cases.
The LangSmith pricing page lists a free Developer plan with one seat and up to 5,000 base traces a month. Plus costs $39 per seat a month with up to 10,000 base traces, then pay as you go. Seat pricing adds up as reviewers join, so count everyone who will read traces.
4. Datadog Agent Observability: Best When You Already Run Datadog
Datadog sells its LLM monitoring as Agent Observability , covering offline experiments as well as production tracing for AI agents. Datadog says the product catches hallucinations, prompt injection attempts and PII exposure as they happen.
Those safety checks matter because prompt injection is LLM01:2025, the first entry in the OWASP Top 10 for LLM Applications and a core topic in agentic AI security . For a team that already pays for Datadog APM, this is the shortest path to AI traces in production. The AI traces sit next to the service traces, logs and metrics that the on-call team already reads.
Kanerika Service
AI Governance for Production AI
Kanerika sets the access, policy and audit controls that keep AI agents and models safe in production, from prompt-injection defenses to evidence for auditors.
Explore AI Governance Cost visibility, however, is the weak spot. Datadog’s price list sets Agent Observability at $160 a month for the first 100,000 LLM spans, billed annually with 15-day retention. That charge lands inside an existing Datadog bill, so it is easy to lose sight of what AI monitoring alone costs. Leaving later also means re-instrumenting, unless your traces follow OpenTelemetry from the start.
5. Braintrust: Best for Evaluation-Led Release Gating
Braintrust starts from evaluation rather than dashboards. Teams build scored test sets, run them against every prompt or model change and block a release if quality drops. Production traces then feed back into those test sets.
That workflow suits teams that ship weekly and want a hard quality gate in CI. It also fits a wider plan to evaluate AI agents before and after launch. Teams that mainly want live operational monitoring may find it lighter on infrastructure context than an APM-based tool. A failed gate also records which test cases broke, which helps when someone asks why a release was held back.
According to Braintrust’s pricing page , Starter is $0 and Pro is $249 a month. Enterprise pricing is custom and also adds on-prem or hosted deployment plus role-based access control.
6. Comet Opik: Best Free Self-Hosted Platform
Opik is Comet’s open-source platform for debugging, evaluating and monitoring LLM applications, RAG systems and agent workflows. You trace a workflow, then score each run with LLM-as-judge metrics and turn failed runs into a dataset for the next round of tests.
The Opik repository is licensed under Apache-2.0, and the full platform is free to self-host. Apache-2.0 is a permissive OSI license, so legal review tends to be simpler than for source-available terms.
Teams that would rather not run servers can start on Comet’s cloud plans , with a free tier and Pro Cloud at $19 a month. On the self-hosted route the license is free but the operations are not. Plan storage, upgrades and access control before production traffic arrives, and start with one workflow before widening once the team trusts the scores.
Kanerika Service
AI Model Development and MLOps
Kanerika builds, deploys and monitors AI models in production, with model evaluation, testing and MLOps practices that keep quality visible after launch.
Explore AI Model Services 7. Fiddler AI: Best for Regulated Enterprises
Fiddler AI describes itself as the control plane for enterprise AI agents. The platform combines guardrails, ML observability and governance in one place.
That mix especially suits banks, insurers and health systems. Many of them run credit or claims models next to newer LLM assistants, and they must explain both to auditors. One governance team can watch model performance, enforce policies and keep the evidence together, which pairs well with disciplined ML model management .
Small teams may not need a control plane yet, since it asks for more setup and process than a developer tracing tool. Pricing is not public, so ask for a quote that covers both ML and LLM volumes. Ask as well how ML and LLM alerts are routed, so each one reaches the right team.
8. Galileo: Best for Turning Evals Into Guardrails
Galileo builds its product around one idea, which is that the evaluations you write before launch should become the guardrails that run after it. On its website , Galileo says its Luna models monitor 100% of traffic at 96% lower cost, with more than 20 ready-made evaluations. Still, treat those as vendor figures and test them on your own data.
Its research roots are public too, including the ChainPoll hallucination-detection method described in a 2023 paper by Friel and Sanyal . That makes Galileo a strong pick when LLM hallucination is the main risk. A typical case is a retrieval-augmented generation pipeline answering from large document sets.
Blocking guardrails also add latency and need careful tuning. A guardrail that blocks good answers costs trust as fast as one that misses bad ones. Start in monitor-only mode, then switch to blocking once false positives are low. Pricing is quoted by the vendor.
Watch on YouTube
LLM Hallucination in Enterprise Data: The Semantic Layer Fix
Kanerika explains why LLMs hallucinate on enterprise data and how a semantic layer grounds answers, the same failure that guardrails and evaluators are built to catch.
9. W&B Weave: Best for Teams Already on Weights & Biases
Weave is the LLM tracing and evaluation toolkit from Weights & Biases. Day to day, you decorate a function with @weave.op() to trace its calls, then score responses with LLM judges and custom scorers. If your data scientists already track experiments in W&B, Weave keeps LLM traces in the same workspace.
Ownership is the question to raise before you sign. CoreWeave completed its acquisition of Weights & Biases on May 5, 2025. CoreWeave’s documentation names CoreWeave Agent Lens as the successor to Weave, with nothing to migrate.
Ask which product the roadmap favors, and raise data residency and contract terms with the new owner. For teams outside the W&B ecosystem the case is weaker, and Langfuse or Opik may fit better.
10. Microsoft Foundry Observability: Best for Azure-Native AI
Teams building on Azure get observability inside Microsoft Foundry. According to Microsoft Learn , tracing is built on OpenTelemetry and integrated with Azure Monitor Application Insights.
Foundry also ships evaluators for groundedness and relevance, among others. Before deployment, an AI red teaming agent uses Microsoft’s PyRIT framework to probe for safety and security weaknesses. Together those pieces cover the loop from testing to production for Azure-first teams.
If any part of the agent runs outside Azure, a neutral trace store may serve you better. Our guide to Azure monitoring tools covers the wider Azure Monitor setup.
Also Worth a Look
A few more tools belong on a long list, depending on what you already run.
New Relic AI Monitoring adds AI traces for teams already standardized on New Relic APM.Dynatrace extends its full-stack observability to AI workloads in large hybrid estates.Evidently AI is an open-source library for ML and LLM evaluation and data drift reports.MLflow tracing suits teams that already manage models in MLflow, including many Databricks users.Helicone, Pydantic Logfire and TruLens are further names to check for tracing depth and license terms before a pilot.
Vendor Shake-Out, 2024 to 2026: Who Owns Your Observability Tool Now
Buying one of these AI observability tools is a multi-year bet, and several vendors in this market have changed hands. The table lists only changes we could confirm at a primary source.
Vendor What changed When
TruEra Snowflake announced an agreement to acquire TruEra to bring LLM and ML observability into its data cloud May 22, 2024 Weights & Biases CoreWeave completed the acquisition, and its documentation names CoreWeave Agent Lens as Weave’s successor, with nothing to migrate May 5, 2025 Langfuse Joined ClickHouse, while core features stay open source under the existing MIT license January 16, 2026
None of these changes makes a tool a bad choice. Each one can still shift pricing, roadmap or support over a three-year contract, so the practical defense is portability.
The OpenTelemetry semantic conventions for generative AI define standard names for model calls and agent spans. A model span is named for the operation and the model, such as chat gpt-6.1-sol. If your traces follow these names, moving to another backend means changing an exporter rather than rewriting instrumentation.
OpenTelemetry Portability: What a Trace Must Keep When It Moves
A vendor that says it accepts OTLP has not yet proved your traces are portable. OTLP is only the transport, so a backend can accept the data while still flattening or dropping the fields that make it useful. The test that matters is whether a trace exported from one tool still answers the same questions in a second tool.
Check that these six fields survive the move:
Parent-child span links. Retrieval, tool and model spans must still nest under one request, or you lose the step that failed.Model and provider. gen_ai.request.model and gen_ai.provider.name must arrive intact, so a model swap still shows up as a change you can compare.Token counts. Input and output tokens per span must match, so cost reports agree across tools.Tool-call attributes. Tool name, masked arguments and result status must be there for every call, MCP calls included.Evaluation scores. Groundedness and policy scores must travel with the trace rather than sit only in one vendor’s own tables.Session id. The conversation identifier must survive, so a multi-turn failure can be replayed in order.
Run this check during the pilot, not at renewal. Send the same week of traces to two backends from one collector, then compare the six fields side by side. Keep one caveat in mind, though. The GenAI conventions still carry the status Development as of November 3, 2026, so attribute names can change between releases.
How to Choose the Right AI Observability Tool
The right AI observability tool is usually the one that fits the stack and risk profile you already have. Feature lists look alike across vendors, so start from your own constraints. Five questions, which spell TRACE, narrow a long list to two or three tools worth a pilot.
Five Questions That Narrow the Shortlist
Tech stack. Which framework, cloud and APM do you already run? A LangGraph team on Datadog has a different shortlist than a Python team on Azure.Regulation. Do auditors need evidence of model behavior? A bank’s credit-decision assistant needs retention, access control and records that support explainable AI , so Fiddler or Arize moves up.Agent complexity. Is it one model call or a chain of tools? Agents that plan and call several tools need span-level tracing across every step.Cost and team size. How many people will read traces, and how much traffic flows? Five engineers on seat pricing cost far less than fifty reviewers.Evaluation or monitoring. Is your main risk shipping a bad release, or drifting after launch? Braintrust leans toward pre-release gates, while Arize leans toward live monitoring.
Answer these before you book any demos, and most shortlists shrink to two names. If the fifth question points to a platform that owns the release cycle, also compare the LLMOps observability platforms that run the evaluation loop .
Which Tool Fits Which Situation
The table maps common enterprise situations to a first pick, such as Langfuse for strict data residency. Treat it as a starting point for a pilot, then confirm it with your own traffic.
Your situation Start with Why
Data must stay inside your network Langfuse or Comet Opik Full self-hosting under open licenses Agents built on LangChain or LangGraph LangSmith Native tracing, so setup takes very little code Datadog already runs your APM Datadog Agent Observability AI traces sit beside service traces and logs Bank, insurer or health system with ML and LLMs Fiddler AI or Arize ML monitoring plus LLM evaluation and guardrails AI built on Azure and Microsoft Foundry Foundry observability OpenTelemetry tracing into Application Insights Weekly releases gated by quality scores Braintrust or Galileo Evaluations run before and after each deploy
Two list-price thresholds, checked on November 3, 2026, sharpen these picks. Langfuse Hobby is free up to 50,000 units a month and two users, so a small pilot needs no budget line. LangSmith’s free Developer plan covers one seat and 5,000 base traces, while three reviewers on Plus cost 3 × $39, or $117 a month.
When neither threshold decides it, let your risk owner break the tie. If security must approve every new data processor, start self-hosted. If speed to production matters more, start on the managed tier of the tool your stack already favors.
A table like this ages quickly, because stacks change. If you move from LangChain to your own framework, or from one cloud to two, revisit the pick before renewal.
AI Assessment
How Ready Is Your AI for Production Monitoring?
Kanerika’s AI Maturity Assessment scores your data, processes and skills, so you can see where observability and governance gaps sit before you scale AI.
Start Your AI Assessment → Worked Example: A Customer-Service Agent on Azure OpenAI
Take a retail company running a customer-service agent on Azure OpenAI. The agent answers order questions, checks policy documents and can open a return. Here is how we would set up observability for it, step by step.
Trace every turn. Record one trace per customer message, with spans for retrieval, each tool call and each model call. Mask names, emails and card numbers before export.Pick the evaluations. Score groundedness and relevance with Foundry’s built-in evaluators. Add a custom check that compares quoted policies against the current policy library.Set starting thresholds. As an assumption to tune after two weeks of baseline, alert when more than 5% of sampled answers fail groundedness in an hour. Also alert when p95 latency stays above 8 seconds for 15 minutes.Watch cost per outcome. Track cost per resolved conversation. As a starting point, alert when it rises 25% above its seven-day average.Name the owners. Quality alerts go to the ML engineer on call, while cost alerts go to the product owner.Choose the tool. Foundry observability fits, because the whole stack runs on Azure. Export traces in OpenTelemetry format, so Langfuse or Phoenix could be added later.
With this setup, the retired-policy answer from the opening scenario would trip the policy check within the hour. The thresholds above are starting assumptions, not benchmarks, so record what they catch in the first two weeks and tune them.
Six Mistakes to Avoid
Even a good tool fails when the rollout skips the basics. These six mistakes show up again and again in enterprise pilots.
Logging raw PII. Prompts and responses carry customer data, so mask it before it leaves the application.Sampling away the failures. Random sampling drops rare errors, so always keep traces with low scores or user complaints.Skipping the baseline. Without a recorded eval baseline, no one can tell whether a release made things worse.Tracing only the model call. Retrieval and tool calls cause many wrong answers, so trace them too.Locking in without an exit. A tool whose exports drop span links or eval scores makes the next migration expensive.Alerts with no owner. An alert that pages nobody is just a log line.
Each of these mistakes treats observability as a dashboard project rather than a control on what the AI system is allowed to do. The episode below looks at the same problem from the risk side, asking how enterprises keep AI agents from acting outside their limits.
What AI Observability Costs
Pricing for AI observability tools follows five models, and each one grows with a different part of your business. Seats grow with the team, while usage grows with traffic. The table shows which is which, using list prices as of November 3, 2026.
Pricing model How it is billed Examples Watch out for
Per seat A monthly fee for each user LangSmith Plus at $39 per seat Costs climb as reviewers and analysts join Per usage unit An included allowance, then a rate per block Langfuse at $8 per extra 100,000 units Verbose agent traces and traffic spikes Flat platform tier A fixed monthly fee with plan limits Braintrust Pro at $249, Langfuse Pro at $199 Limits that force an upgrade mid-year Inside a platform contract Added to an APM or cloud bill Datadog, Microsoft Foundry AI spend hidden inside a bigger invoice Self-hosted open source No license fee Langfuse, Phoenix, Opik Servers, storage, upgrades and on-call time
A Worked Monthly Estimate
Assume a team of five engineers sends about one million billable units a month. These figures use list prices as of November 3, 2026, before any volume discount or tax.
On Langfuse Core, the $29 base covers the first 100,000 units. The other 900,000 units cost 9 × $8, or $72, so the total is $101 a month. Langfuse says the rate gets lower with volume, so ask for the price at your expected traffic.
On LangSmith Plus, five seats cost 5 × $39, or $195 a month. That covers up to 10,000 base traces, but anything above runs pay as you go. Braintrust Pro, by comparison, starts at $249 a month for the platform.
Units and traces are counted differently, so these numbers do not compare like for like. Run a two-week pilot, measure your real volume, then ask each vendor for a quote on that number.
Cost per Successful Task
The monthly bill tells you what observability costs, but not whether the AI system is worth running. A better unit is cost per successful task, which counts only the work the system got right.
Cost per successful task = (model inference + evaluation calls + observability tool + storage) ÷ tasks completed correctly
Each input comes from a different place. Inference and evaluation costs come from token counts on the spans. The tool fee comes from the vendor invoice, while storage comes from your retention settings. The denominator comes from your quality scores, which is why evaluation has to be in place before this number means anything.
Here is an illustrative calculation using the Langfuse figure above. Suppose the support agent handles 20,000 requests in a month and evaluation judges 17,000 of them correct. Inference and evaluation calls cost $1,900, while storage costs $100. Adding the $101 Langfuse bill gives $2,101, or about $0.12 per successful task.
This is the same number as the KPI cost per resolved request in the Kanerika section below. Track it weekly, because a cheaper model that fails more often can raise it even while the token bill falls.
Enterprise Requirements Checklist
Enterprise buyers need more than good traces, because the traces themselves hold sensitive data. Before a pilot turns into a contract, put these ten questions to every vendor on the shortlist.
Requirement Why it matters Ask the vendor
SSO and SAML Access follows your identity provider, so leavers lose access at once Which identity providers do you support on our plan? Role-based access Support staff should not read every prompt Can we restrict traces by project and role? Data residency Traces often hold personal data under regional rules Where are traces stored, and can we choose the region? Self-host or VPC Some data cannot leave your network Do you support private deployment, and on which plan? PII masking Prompts carry names, emails and account numbers Is masking applied before data leaves our application? Redaction before ingestion Hashed or deleted attributes never reach the vendor in raw form Will you accept traces that our own collector has hashed or redacted, and can we set the region and retention per project? Retention and export Audits and migrations need old traces How long do you keep data, and in what export format? Audit logs Regulators ask who viewed or changed what Do you log user actions inside the platform? OpenTelemetry export Portable traces protect you if the vendor changes hands Can we export traces in OpenTelemetry format with span links and eval scores intact? SLA and support Observability outages hide production failures What uptime and response times do you commit to?
Redaction, Region and Retention
The redaction row deserves the most attention for regulated data. The OpenTelemetry Collector’s attributes processor can delete an attribute or replace it with a SHA2-256 hash inside your network, as Stage 1 below shows. Pair that with a fixed data region and a retention period your legal team has signed off.
A vendor that answers all ten clearly, in writing, is ready for enterprise work. Vague answers on residency or export are a signal to keep looking.
Checklist
Agentic AI Readiness Checklist
Check data, governance, monitoring and ownership readiness before your AI agents reach production traffic.
Get the Checklist → Implementing AI Observability in Four Stages
Rolling out observability works best in stages, each with a named owner and a clear exit test. Moving on before the exit test passes is how teams end up with dashboards nobody trusts.
Stage 1: Instrument Every Request
Platform engineering owns this stage, because it controls the application code. Add tracing to every production request, covering retrieval, tool calls and model calls, with PII masked at the source. The exit test is simple: every request produces a trace with model, provider, prompt version, tokens and latency.
Record the model id, provider, prompt version and fallback route as span attributes from the start. When you swap models or traffic falls back to a second route, the change then shows up in the traces as something you can compare.
In code, that means at least two spans per turn. The sketch below uses the OpenTelemetry Python API with attribute names from the GenAI inference and retrieval conventions, which still carry the status Development. Here index and llm stand in for your own retriever and model client, and the app.* keys are custom.
from opentelemetry import trace
tracer = trace.get_tracer("support-agent")
def answer(question, index, llm, prompt_version="refunds-v7", route="primary"):
# Retrieval span: which documents grounded the answer
with tracer.start_as_current_span("retrieval") as span:
span.set_attribute("gen_ai.operation.name", "retrieval")
span.set_attribute("app.retrieval.index", "policy-index")
span.set_attribute("gen_ai.retrieval.top_k", 3)
docs = index.search(question, top_k=3)
span.set_attribute("app.policy.doc_ids", [d.id for d in docs])
span.set_attribute("app.policy.any_retired", any(d.retired for d in docs))
# Model span: model, provider, prompt version, route and token counts
# use your deployed model name
with tracer.start_as_current_span("chat gpt-6.1-sol") as span:
span.set_attribute("gen_ai.operation.name", "chat")
span.set_attribute("gen_ai.provider.name", "azure.ai.openai")
span.set_attribute("gen_ai.request.model", "gpt-6.1-sol")
span.set_attribute("gen_ai.prompt.version", prompt_version)
span.set_attribute("app.route.name", route)
reply = llm.complete(question, docs)
span.set_attribute("gen_ai.usage.input_tokens", reply.input_tokens)
span.set_attribute("gen_ai.usage.output_tokens", reply.output_tokens)
return reply.textIn the opening scenario every call returned 200 OK, so latency and error dashboards stayed green. With app.policy.any_retired on the retrieval span, one trace query finds every answer grounded in a retired document. The same flag can drive the Stage 3 alert.
Watch on YouTube
What Does It Take to Deploy an Enterprise LLM?
A practical conversation on what an enterprise LLM deployment needs beyond the model itself, from evaluation and monitoring to ownership in production.
A second masking layer can run in an OpenTelemetry Collector inside your network, before any trace leaves it. This config uses the attributes processor to hash the user email and delete raw message content. The exporter block points at the Langfuse cloud endpoint for OpenTelemetry data.
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
processors:
attributes/mask:
actions:
- key: user.email
action: hash
- key: gen_ai.input.messages
action: delete
- key: gen_ai.output.messages
action: delete
batch: {}
exporters:
otlphttp/langfuse:
endpoint: "https://cloud.langfuse.com/api/public/otel"
headers:
Authorization: "Basic ${env:LANGFUSE_AUTH}"
x-langfuse-ingestion-version: "4"
service:
pipelines:
traces:
receivers: [otlp]
processors: [attributes/mask, batch]
exporters: [otlphttp/langfuse]Changing backends then means editing the exporter block, not the application code. That is the portability test from the shake-out section, put into practice.
Stage 2: Define Failure Modes
Product owners and domain experts lead here, since they know what a wrong answer looks like. Write down the top failure modes, such as outdated policy, wrong product or unsafe advice, and collect real examples of each.
For a retrieval-based assistant, separate retrieval failures from generation failures, because the fixes differ. Our comparison of RAG versus a plain LLM shows where each kind of error tends to start. The stage ends when every failure mode has a written definition and a labeled example set.
Stage 3: Layer Evaluation
ML engineering and QA then turn those failure modes into automated checks. Offline evaluations run in CI before each release, while online evaluators score a sample of live traffic. Our LLM evaluation framework covers metric choice and test design in depth.
Sample with intent rather than at random. Keep every trace with a low score, a user complaint or a fallback route, then sample the rest. The exit test is a recorded baseline score for each check. Store it next to the prompt version, so a later regression traces back to a specific change.
Stage 4: Connect to Governance
Finally, risk, compliance and platform teams close the loop together. Set retention rules, access roles and alert owners, then agree on how evidence gets exported for audits. Give each export a named owner, so an audit request never lands on whoever happens to be on call. The stage is done when an auditor’s request can be answered from the tool without a custom script.
Observability as Audit Evidence: EU AI Act Logging and NIST AI RMF
The traces engineers use to debug are also the most practical evidence when regulators or internal auditors ask how an AI system behaved. Two frameworks make that link explicit.
Article 12 of the EU AI Act sets a logging duty. High-risk AI systems “shall technically allow for the automatic recording of events (logs) over the lifetime of the system,” in the Act’s words. The AI Act Explorer lists application dates of December 2, 2027 for Annex III systems and August 2, 2028 for Annex I systems.
Trace retention and export settings are how you meet that duty in practice. Set retention to the period your legal team defines, and make sure exports keep the six fields from the portability test.
In the United States, the NIST AI Risk Management Framework was released on January 26, 2023, and it is voluntary. Its focus on measuring and managing AI risk maps directly to evaluation scores and drift alerts, the measurable side of responsible AI . NIST added a Generative AI Profile, NIST AI 600-1, on July 26, 2024.
Observability supplies the evidence, while governance decides what to do with it. For the policy side, see our overview of AI governance . Our guide to AI regulation covers the wider legal picture.
How Kanerika Helps Enterprises Run Observable AI
Kanerika builds and runs AI systems for enterprises, and we treat observability as part of the build rather than an add-on. Our agentic AI services put tracing, evaluation and approval gates into agents from the first release. Those traces then become evidence an auditor can use, through our AI governance services .
Our Delivery Approach
Every engagement follows four stages, so nothing depends on a single tool.
Assess. We review the stack, the risk profile and which data the traces will carry, including personal and regulated fields.Instrument. We add OpenTelemetry tracing and mask sensitive attributes at the collector, backed by our data governance services where the masking policy needs work.Baseline. We set evaluation baselines and alert thresholds with your product owners, so every alert has an owner.Hand over. We deliver runbooks and audit evidence that your own team can maintain.
The masking pattern is one we have applied before. On an AI DataOps platform we added correlation-ID tracing with masking built in for sensitive data, as the AI DataOps case study describes.
For tool selection, we recommend a two-week pilot on real traffic. Feed two shortlisted tools from the same OpenTelemetry collector. Then compare trace completeness and how long each takes to surface a seeded failure, such as a planted retired-policy document.
Case Study: 65% Self-Service Resolution with an AI Support Agent
One Kanerika project shows why confidence is a signal worth tracing. We built an AI member support agent that integrates with knowledge bases and Zendesk. According to the AI member support agent case study , it can “instantly resolve 65% of member queries.”
The agent does not try to answer everything, though. It routes complex cases to live executives when confidence is low, so the handoff is part of the design rather than a failure.
That design maps straight onto the opening scenario. The confidence score and the handoff are the two signals an observability tool must record, trend and alert on. A falling confidence trend or a rising handoff rate can be an early sign that a policy document or a retrieval index has gone stale.
Case Study
65% of Member Queries Resolved Instantly by an AI Support Agent
Kanerika built an AI member support agent that integrates with knowledge bases and Zendesk, resolves 65% of member queries instantly and routes complex cases to live executives when confidence is low.
Read the Case Study → Three KPIs to Track
Three numbers tell you whether observability is working, and each one has an owner.
Mean time to detect a quality regression , measured from release to first alert.Cost per resolved request , the cost per successful task from the cost section, so model spend ties to outcomes rather than calls.Share of traffic evaluated , which shows how much of production you can actually vouch for.
Conclusion: Which AI Observability Tool Should You Pick?
Start with your stack, then your risk. Langfuse and Opik suit teams that must self-host, LangSmith suits LangChain builders and Datadog suits teams that already live in Datadog. Arize and Fiddler fit regulated estates with both ML and LLMs, while Foundry observability fits Azure.
Whichever you pick, insist on OpenTelemetry export and test that the six portability fields survive. Then run a two-week pilot on real traffic. If you want help scoping that pilot, talk to Kanerika’s AI team .
The best AI observability tool for your enterprise team is the one that catches the wrong answer on your own traffic, before your customers do.
Frequently Asked Questions
What is AI observability and how is it different from AI monitoring?
AI monitoring tracks whether a model call succeeded, how long it took and what it cost. AI observability records what the model saw, did and said, including the prompt, the retrieved context, every tool call and a quality score. That record explains why an answer was wrong even when the request returned 200 OK.
Which AI observability tool is best for teams just getting started?
Start with a tool that has a free tier and fits your framework. Langfuse has a free Hobby plan with 50,000 units a month and can be self-hosted later. LangSmith gives LangChain users a free Developer seat, and Comet Opik is free to self-host. Trace one use case for two weeks, then decide.
Is AI observability required for regulatory compliance?
For high-risk systems under the EU AI Act, Article 12 requires automatic recording of events over the system’s lifetime, with application dates of December 2, 2027 for Annex III and August 2, 2028 for Annex I. The NIST AI Risk Management Framework is voluntary, but its measure and manage functions map directly to traces and evaluation scores.
How much do enterprise AI observability tools cost?
List prices checked on November 3, 2026 start free. Langfuse Core costs $29 a month and Enterprise $2,499, LangSmith Plus is $39 per seat a month and Braintrust Pro is $249 a month. Datadog Agent Observability starts at $160 a month for 100,000 LLM spans, Microsoft Foundry bills through Azure, and Fiddler and Galileo quote on request, so price a pilot on your real volume.
What is the best AI observability tool in 2026?
No single tool wins for every team. Langfuse or Opik suit teams that must self-host, LangSmith suits LangChain builders and Datadog suits teams already on Datadog. Arize and Fiddler fit regulated estates with both ML and LLMs, and Foundry observability fits Azure. Pick by stack and risk, then confirm the choice with a two-week pilot.
Does AI observability replace application performance monitoring?
No, the two work together. Application performance monitoring tells you the system is up by tracking requests, latency and errors across services. AI observability tells you the system is right by recording prompts, context, tool calls and quality scores. Many teams send both kinds of trace through one OpenTelemetry collector so on-call engineers see the whole request.
Do I need a separate AI observability tool if I already use Datadog or New Relic?
Often you can start with what you already have. Datadog Agent Observability and New Relic AI Monitoring add AI traces next to existing service data. A separate tool makes sense when you need deeper evaluation, self-hosting or open-source licensing, or when model teams want their own workspace for prompts, datasets and test sets.
Is Langfuse better than LangSmith?
It depends on your stack and your data rules. Langfuse suits teams that want to self-host an open-source platform, with a free Hobby tier and Core at $29 a month. LangSmith suits teams building on LangChain or LangGraph, with a free Developer seat and Plus at $39 per seat a month. Pilot both on identical traffic.
How do you attribute LLM cost to teams or requests?
Tag every span with a team, feature and user id, then multiply input and output token counts by each model’s price and sum by tag. Add evaluation calls, the observability tool fee and storage to get the full picture. Dividing that total by tasks completed correctly gives cost per successful task, the number worth tracking weekly.
Are there free or open-source AI observability tools?
Yes. Langfuse has an open-source core that you can self-host, Comet Opik is licensed under Apache-2.0 and free to self-host, and Arize Phoenix is source-available under the Elastic License 2.0. The license costs nothing, but your team still pays for servers, storage, upgrades and on-call time, so budget for operations before production traffic.
Why does OpenTelemetry support matter for AI observability?
OpenTelemetry is not strictly required, but it is strongly advised. It gives traces standard field names, so you can change backends by editing an exporter instead of rewriting code. Accepting OTLP is not enough on its own, so test that span links, model ids, token counts, tool calls, eval scores and session ids survive the move.
What metrics should an AI observability platform track?
Track five signals for every request. These are traces with a span per step, quality scores such as groundedness and relevance, tokens and cost, latency per span, and safety events such as prompt injection attempts or personal data in prompts. Roll them up into cost per resolved request and mean time to detect a quality regression.
How do AI observability tools evaluate guardrails and hallucinations?
They score each answer against the retrieved context, using groundedness checks, LLM judges or small dedicated evaluator models. Galileo, for example, says its Luna models monitor 100% of traffic at 96% lower cost, which is a vendor figure worth testing yourself. Start guardrails in monitor-only mode and switch to blocking once false positives are low.
What is the difference between AI observability and data observability?
Data observability watches pipelines, freshness, schema and volume, so it warns you when the data feeding an AI system arrives late or broken. AI observability watches what the model does with that data, including prompts, tool calls, answers and quality scores. Production AI usually needs both, because a stale table and a wrong answer look alike to users.
Which AI observability tools support multi-agent monitoring?
LangSmith traces each step of a LangGraph workflow as its own span, and Datadog Agent Observability covers agents in production next to service traces. Fiddler positions itself as a control plane for enterprise AI agents, while Langfuse, Arize Phoenix and Opik also trace agent workflows. Check that every agent handoff and tool call sits inside one trace.
How do AI observability tools handle RAG pipeline monitoring?
They trace the retrieval step as its own span, so you can see which documents fed each answer. Evaluators then score groundedness and relevance against that retrieved context, and Microsoft Foundry ships built-in evaluators for both. When scores drop, the trace shows whether retrieval pulled stale content or the model ignored good content that was available.
How long does it take to implement an AI observability platform?
Basic tracing on one application can run within days when the team uses a supported framework. A full rollout takes longer, because it moves through four stages of instrumenting requests, defining failure modes, building evaluations and connecting results to governance. Plan a two-week pilot first, then expand stage by stage with a clear exit test for each.
