TL;DR
AI red teaming is structured adversarial testing that tries to make an AI system behave in ways its owners never intended. It tests the deployed system, so prompts, retrieval and agent tools all count. Penetration testing does not replace it, because AI systems fail probabilistically. OWASP, NIST and MITRE ATLAS give teams a shared vocabulary for findings. The EU AI Act now requires documented adversarial testing from some model providers. Red teaming works as one part of a wider AI testing program.
Key Takeaways AI red teaming targets the whole deployed system, not the model weights in isolation. Penetration testing and AI red teaming answer different questions and need different pass criteria. OWASP, NIST AI RMF and MITRE ATLAS supply the taxonomy that makes findings comparable. EU AI Act Article 55(1)(a) requires documented adversarial testing from systemic-risk model providers. CVSS scoring does not translate to AI harms, so reports need an autonomy-based severity rubric. Independent research finds the term has narrowed toward prompt hacking, leaving other risks untested. AI Red Teaming as a Methodology for Testing Enterprise AI Systems Microsoft’s AI red team published lessons from testing more than 100 generative AI products across more than 80 operations in a January 2025 paper . That volume at one vendor says something plain. Adversarial testing has moved out of research and into the release path.
Security leaders still have to decide who runs the testing, how often, and what a finding must contain before engineering acts. Vendor definition pages answer none of that. In this article, we’ll cover what AI red teaming tests, how an engagement runs, the frameworks and rules shaping it, the severity model, the metrics, and where it belongs in a wider testing program.
What AI Red Teaming Is and What It Actually Tests A Working Definition AI red teaming is structured adversarial testing that tries to make a deployed AI system produce harmful, leaked or unauthorized behavior. Testers work against the running application, including its prompts, retrieval sources and tools. The output is a set of reproducible failures with evidence, severity and a named fix owner.
The discipline takes its name from security red teaming and about half its method. Microsoft’s second lesson states that you do not have to compute gradients to break an AI system. Natural language and patience are often enough, which widens the pool of plausible attackers.
What Gets Tested, From the Model to the Agent OWASP’s GenAI Red Teaming Guide , published on 22 January 2025, splits the work into four areas. Those four make a usable scope statement, and they stop a test collapsing into prompt play.
Model evaluation, covering how the model itself behaves under adversarial input. Implementation testing, covering guardrails, system prompts and the application wrapper. Infrastructure assessment, covering the serving stack, pipelines and supporting services. Runtime behavior analysis, covering what the live system does once it is connected. A mature AI security assessment covers all four. Teams that test only the first area report clean results and still ship an exploitable product.
Red Teaming, Evaluation, and Benchmarking Are Not the Same Thing Microsoft’s third lesson states that “AI red teaming is not safety benchmarking”. A benchmark measures average behavior on a fixed dataset. A red team hunts the worst behavior an adversary can force.
Evaluation sits between the two. An LLM evaluation framework scores quality and safety on repeatable suites, and AI agent evaluation extends that to multi-step tasks. Red teaming feeds both, because every confirmed attack becomes a regression case.
Why Penetration Testing Does Not Cover AI Systems A penetration test hunts a defect that either exists or does not. An AI red team test hunts a behavior that appears in some attempts and vanishes in others. That difference changes scope, pass criteria, tooling and retest cadence.
Conventional testing still has a job here. The API gateway, the vector store and the credential broker all need it. An agentic AI vulnerability assessment runs alongside a standard engagement rather than instead of one.
Table 1: Traditional penetration testing compared with AI red teaming across seven dimensions
Dimension Traditional Penetration Testing AI Red Teaming Scope Network, hosts, application code, identity Model, prompts, retrieval, tools, connectors, runtime behavior Primary inputs Crafted packets, payloads, credentials Natural language, documents, images, tool responses Failure modes Deterministic, reproducible on demand Probabilistic, sensitive to phrasing and model version Pass criteria Vulnerability closed and verified once Attack success rate below an agreed threshold across repeated trials Tooling Scanners, exploit frameworks, proxies Adversarial prompt frameworks, attack orchestration, scorers Who runs it Offensive security specialists Mixed security, ML engineering, domain and policy experts Cadence Annual or on major release Continuous, with retests bound to model, tool and data changes
Kanerika Service
AI Governance Designed Into Delivery, Not Bolted On
Kanerika runs governance and adversarial testing alongside build work on Microsoft Purview through the kanSuite services program.
Explore AI Governance
The AI Attack Surface Enterprises Have to Test The surface widens every time a system gains a document store, a tool or a connector. OWASP’s Top 10 for LLM Applications, 2025 edition gives each category a stable identifier. OWASP has announced a 2026 update, and the published list remains the 2025 one.
Prompt-Layer Attacks Prompt injection sits at LLM01 and remains the most reliable way into a generative application. System prompt leakage sits at LLM07, handing an attacker the guardrail wording they need to defeat. Improper output handling at LLM05 turns model text into executable content downstream, which an LLM security guide treats as an application flaw.
Data and Privacy Attacks Sensitive information disclosure at LLM02 covers training data, system context and anything the model can reach at inference. Data and model poisoning at LLM04 covers contaminated fine-tuning sets and tainted upstream artifacts. Testing here overlaps with AI data leakage controls and with the supply chain category at LLM03.
RAG and Retrieval Attacks Vector and embedding weaknesses sit at LLM08. An attacker who can write into an indexed source can plant instructions the retriever later treats as trusted context. Systems built on agentic RAG need the index tested with poisoned documents, not only clean queries.
Agent, Tool-Calling and Connector Attacks Excessive agency at LLM06 describes a model that can take actions beyond what the task requires. OWASP’s Top 10 Risks and Mitigations for Agentic AI Security arrived in December 2025. It names agent behavior hijacking, tool misuse, identity and privilege abuse, goal hijacking, memory poisoning and rogue autonomous behaviors .
Each one needs a test that exercises actions rather than answers. Teams designing AI agent architecture should assume a hostile tool response at least once. Every connector exposed through the Model Context Protocol is its own trust boundary, and scoped AI access control decides how far a hijacked agent gets.
Attacks on Models That Are Not Language Models Classifiers, recommenders, forecasting models and vision systems carry their own adversarial history. MITRE ATLAS documents adversary tactics and techniques against AI systems of many kinds, with real-world case studies attached. Model lineage tracking makes poisoning and theft scenarios testable rather than theoretical.
How an AI Red Team Engagement Runs, Phase by Phase A credible engagement follows five phases, and the first decides the value of the other four. Microsoft’s opening lesson puts understanding what the system does and where it is applied ahead of any attack work. Skipping it produces generic jailbreak reports that no product owner can act on.
Threat model and scope. Map the use case, the data the system can reach and the actions it can take, then write down the harms that matter to this business.Access and rules of engagement. Agree environment, model access level, accounts, data handling and stop conditions in writing before testing starts.Manual adversarial probing. Testers with domain and policy knowledge hunt failures that depend on context a generic suite will never construct.Automated campaign expansion. Confirmed attacks become seeds for automated runs across variants, languages and repeated trials, which produces rates rather than anecdotes.Validate, remediate and retest. Each finding gets a reproduction, an owner, a fix and a retest at the same trial count, then joins the permanent regression suite.The Frameworks and Standards Worth Anchoring To Four reference points carry most enterprise programs. OWASP supplies the vulnerability taxonomy, NIST supplies the governance structure, and MITRE ATLAS supplies the adversary knowledge base. The EU AI Act sets a legal floor for some providers.
The 2025 OWASP list names LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data and Model Poisoning and LLM05 Improper Output Handling. It continues with LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation and LLM10 Unbounded Consumption. Mapping each finding to one identifier makes coverage arguable rather than assumed.
NIST’s AI Risk Management Framework , published as NIST AI 100-1 on 26 January 2023, organizes work under Govern, Map, Measure and Manage. Its Generative AI Profile, NIST AI 600-1, followed on 26 July 2024. Red teaming lives under Measure, and fails without the Govern and Map work that an AI security framework and an AI governance framework put in place.
MITRE ATLAS is modeled on MITRE ATT&CK and catalogues adversary tactics, techniques and documented case studies against AI systems. Testers use it the way network teams use ATT&CK, as a planning grid rather than a checklist.
Case Study
Real-Time Compliance and Risk Detection With an AI Agent
How an AI agent was designed to flag compliance and risk exposure as it happens, with human review kept on the actions that carry consequences.
Read the Case Study
What the EU AI Act Actually Requires, at Article Level Article 55(1)(a) requires providers of general-purpose AI models with systemic risk to “perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing”. That wording makes documented adversarial testing a legal duty for a defined class of provider. The article text is short enough to read in full.
Article 55(1) also covers assessing and mitigating systemic risk, reporting serious incidents to the AI Office, and adequate cybersecurity for the model and its infrastructure. Those duties sit together, so an evidence pack showing testing but no incident process is incomplete.
Dates matter more than most summaries admit. GPAI model obligations including Article 55 started to apply on 2 August 2025. Models placed on the market before that date must reach full compliance by 2 August 2027.
The implementation timeline puts the rest of the Act at 2 August 2026. Annex III high-risk requirements follow on 2 December 2027, and Annex I follows on 2 August 2028.
Enterprises that deploy rather than provide models sit outside Article 55, though assurance questionnaires rarely make that distinction. Teams tracking AI regulation and building an AI compliance evidence trail should treat red team documentation as a retained artifact.
Watch on YouTube
Why AI Agent Projects Fail Before Production?
Manual Red Teaming, Automated Red Teaming, and Where Each Earns Its Place Manual testing finds failures that depend on judgment and domain knowledge. Automation finds failures that depend on volume. Microsoft’s lessons place both in the same program, noting that automation covers more of the risk surface while the human element stays central.
Anthropic’s published account of its own practice , dated 12 June 2024, names six methods in use. They are Policy Vulnerability Testing with external subject-matter experts, Frontier Threats Red Teaming, Multilingual and Multicultural Red Teaming, Automated Red Teaming, Multimodal Red Teaming, and Crowdsourced or Community Red Teaming. Each answers a different question.
Crowdsourcing scales in a way the other two do not. At the DEF CON 31 Generative Red Team Challenge in August 2023, 2,244 participants exchanged 164,208 messages across 17,469 conversations. They worked through 21 challenges drawn from the White House Blueprint for an AI Bill of Rights, 50 minutes each, against models from eight companies.
Tools Enterprise Teams Actually Evaluate Four names come up repeatedly on enterprise shortlists, and they are PyRIT, Garak, Inspect and Promptfoo. PyRIT is Microsoft’s MIT-licensed Python Risk Identification Tool for generative AI, built to help security professionals and engineers identify risks proactively. Microsoft also ships an AI Red Teaming Agent inside Azure AI Foundry.
Judge the other three on the same criteria rather than on feature lists. Attack coverage against the OWASP categories, support for multi-turn and agentic targets, and scorer quality come first. Reproducibility across runs and the shape of the output matter more than the count of built-in probes.
Independent work from CSET Georgetown is worth reading before a shortlist hardens. Its analysis of AI red-teaming design examined 23 resources, of which 18 could be used directly to run tests. It also found AI-powered scorers in over half the evaluation tools surveyed.
A scorer that is itself a model needs its own validation. Ask any vendor how theirs was calibrated.
What an AI Red Team Report Should Contain The Executive Summary a Board Will Read The first page answers three questions in business language. What can an attacker make this system do, how likely is that under realistic conditions, and what happens to the release. Everything technical belongs after that page.
Attack narratives work better than counts. One reproducible path ending in a customer record leaving the estate moves a budget conversation further than forty refusal bypasses.
Why CVSS Does Not Map to AI Harms CVSS assumes a binary vulnerability with a fixed exploitation path and a defined impact on confidentiality, integrity and availability. An AI finding has a success rate, a phrasing dependency and a harm that may be reputational or regulatory. Forcing it into a CVSS vector loses the parts that decide whether the system ships.
Model versions break the mapping further. A fix that drops attack success from 80 percent to 4 percent is real progress with no clean CVSS equivalent. That is why agentic AI risks need their own scoring vocabulary.
A Severity Rubric Built on Autonomy, Privilege and Impact Three variables carry most of the signal. How autonomously the system acts, how much privilege it holds, and what the worst realistic outcome costs. The rubric below is a starting point to tune against a given risk appetite.
Table 2: A severity rubric for AI red team findings based on autonomy, privilege and business impact
Severity Tier Autonomy Level Privilege Held Business Impact Release Action Critical Acts without human review Write access to production systems or funds Regulatory breach, customer data loss, irreversible action Block release, fix and retest before any deployment High Acts with post-hoc review only Read access to regulated or confidential data Disclosure of sensitive records, material misstatement to a customer Block general availability, pilot only with compensating controls Medium Suggests, human approves each action Internal non-sensitive data only Policy-violating content, recoverable process error Ship with mitigation, fix on the next scheduled release Low Read-only response generation Public or synthetic data only Tone, formatting or refusal inconsistency Log, add to regression suite, no release impact
Metrics That Make AI Red Teaming Reportable Four numbers turn a narrative report into something a steering committee can track quarter over quarter. Each needs a fixed measurement protocol. A rate measured at ten trials and a rate measured at a thousand are different claims.
Attack success rate. The share of adversarial attempts that achieve the attacker goal, reported per attack class and per model version at a stated trial count.Risk-surface coverage. The proportion of applicable OWASP categories and ATLAS techniques the engagement actually exercised, which exposes the untested areas nobody discussed.Regression rate across model versions. The share of previously fixed attacks that succeed again after a model, prompt or tool change.Time to remediate. Median days from confirmed finding to verified retest, split by severity tier.These metrics survive contact with production only if the telemetry exists to recompute them. A working LLMOps observability practice supplies the traces that make a regression rate calculable rather than estimated.
Build, Buy, or Run a Bug Bounty Three sourcing models exist and most enterprises end up with a blend. An internal team knows the business context and costs the most to staff. An external engagement brings adversarial range, and it needs a scoping effort buyers often underestimate.
OpenAI’s account of external red teaming , published on 21 November 2024, describes four design steps for a campaign. Composition selection, model access decisions, interface and documentation, and data synthesis sit with the buyer as much as the vendor. A weak brief on any of them produces a weak report.
Bug bounties add breadth at variable quality. They work best once internal testing has cleared the obvious categories. Clear scope, safe-harbour language and triage capacity all have to exist first.
Who Owns AI Red Teaming Inside the Organization Ownership splits three ways in most enterprises. Security owns the method and the threat model, product and ML engineering own the fixes, and risk and compliance own the evidence and the sign-off. That split causes more delay than the testing itself.
A named accountable owner for release decisions resolves the common deadlock, where a finding sits open because nobody can block a launch. Programs with mature AI governance already define that role and put the decision rights in writing before the first engagement.
Skills matter as much as reporting lines. Network testers without ML knowledge will find prompt injections and miss poisoning. ML engineers without security training will find neither reliably.
Where Red Teaming Belongs in the Release Cycle Pre-deployment adversarial testing is now state practice rather than vendor advice. The US and UK AI Safety Institutes published a joint pre-deployment evaluation of OpenAI’s o1 model on 18 December 2024.
The evaluation reported that o1 solved 45 percent of cyber challenges against 35 percent for the best reference model. On software-engineering tasks, o1 reached 50 percent against 67 percent. Their overall conclusion was that “Across the three domains tested, o1 largely demonstrated performance on par with the reference models tested”.
Inside an enterprise, the equivalent is a gate before general availability with a documented severity threshold. Microsoft’s eighth lesson states that the work of securing AI systems will never be complete. That argues for standing capacity rather than an annual event.
Four changes should trigger an automatic retest.
A model version upgrade, including a silent provider-side update. A new tool or function exposed to the agent. A new data source added to the retrieval index. A new connector or downstream integration. Continuous monitoring covers the gaps between gates. AI agent observability catches behavior drift that a point-in-time engagement cannot, and it feeds the next round of test design. Drift nobody catches is how rogue AI behavior reaches production.
Authorization, Legal Exposure and Handling What the Test Produces Written authorization comes before the first prompt. The document names the systems in scope, the accounts and data testers may use, the model access granted, and the stop conditions. Probing a third-party model through its API without reading that vendor’s terms creates exposure no security benefit offsets.
Red teaming produces artifacts an organization would never create on purpose. Successful jailbreaks, extracted personal data, synthesized harmful content and working attack chains all need a retention policy, an encrypted store and a named custodian. Treat the evidence repository as a regulated asset from day one.
Findings against a third-party model belong with that provider under coordinated disclosure. Findings against internal systems feed the AI auditing framework and the audit trail. Decide in advance what gets published and what stays internal.
What AI Red Teaming Looks Like in Regulated Industries Banking and Financial Services Test cases center on unauthorized transaction initiation, disclosure of account data across customer boundaries, and advice that crosses into regulated territory. Systems built for agentic AI in banking hold payment and account privileges, which pushes most findings into the upper severity tiers automatically.
Insurance Adversarial testing focuses on claim manipulation, coverage misstatement and disparate outcomes across protected groups. Deployments of AI agents for insurance claims processing need security probes and fairness probes. A biased denial is a compliance event even when no attacker is involved.
Healthcare Protected health information dominates the threat model, alongside clinically unsafe output and instruction-following that bypasses clinician review. Organizations deploying AI agents for healthcare should run clinical subject-matter experts as testers, since a security specialist cannot judge clinical harm.
Manufacturing Operational technology raises the stakes on any action a model can take. Testing for agentic AI in manufacturing covers poisoned sensor data, manipulated scheduling decisions, and any path from a language interface into a control system. Safety engineering review belongs in scope alongside security review.
Watch on YouTube
Why AI Agents Fail in Production
Red Teaming Alone Is Not a Complete AI Testing Program CSET Georgetown argues in its analysis of the term that AI red teaming has collapsed into prompt hacking. Safety-focused prompt testing, on that reading, neglects model theft, data exfiltration and poisoning. A program that tests only what a chat box will say covers a fraction of the risk.
Anthropic makes a related point about comparability. Its published account cites a “lack of standardized practices” in the field, which makes it “challenging to objectively compare the relative safety of different AI systems”. Two vendors can both claim they red teamed a model and mean entirely different work.
Practical consequences follow for buyers. Ask what scope a report covered against the OWASP four areas, and how many trials produced each rate. Treat a clean report with no methodology section as unevidenced.
Red teaming belongs inside a wider structure with evaluation suites, guardrail testing, supply chain controls, access management and runtime monitoring. No single method catches every failure, which is why generative AI risks need layered assurance.
How Kanerika Red Teams the AI Systems It Builds Kanerika is a Microsoft Solutions Partner for Data and AI, headquartered in Austin, Texas, and one of the earliest Microsoft Purview implementors globally. The security posture behind the delivery work carries ISO 27001, ISO 27701, ISO 9001, SOC 2 Type II and CMMI Level 3 certification, plus GDPR compliance. Recognition includes Microsoft Fabric Featured Partner, Everest Group Major Contender in the Microsoft Azure Services PEAK Matrix 2026, and Forbes America’s Best Startup Employers 2025.
Adversarial testing is built into the IMPACT delivery framework rather than bolted on at the end. Threat modeling happens while the agent design is still on paper, so tool permissions and data scopes get argued before they get coded. Teams running several AI agents in production treat each new connector as a fresh trust boundary with its own test pass.
Named assets carry part of the work. Susan handles PII redaction and sensitive data masking, which shrinks what a successful extraction attack can return. Klara checks contracts against a governance playbook, Mike proofreads quantitative claims inside a document, and Karl works as a data insights agent.
Governance and testing run together through kanSuite, a modular governance services program on Microsoft Purview comprising kanGovern, kanComply and kanGuard, and through AI governance services . The real-time compliance and risk detection case study shows the pattern in delivery. Honest practitioner guidance is that teams over-invest in prompt hardening and under-invest in permission scoping, then find during testing that the agent always reached further than intended.
Talk to Kanerika
Put a Red Team Gate on Your Next AI Release
Kanerika designs the threat model, the test campaign and the release gate around the AI systems you are shipping.
Book a Meeting
Wrapping Up AI red teaming has become the testing method that decides what an enterprise is willing to ship. The frameworks exist, the regulatory floor is set for some providers, and the tooling now supports repeatable campaigns. What stays difficult is the program around the testing, meaning ownership, cadence, severity, evidence and the honesty to report a rate rather than a verdict.
Teams that treat red teaming as a standing capability, anchored to OWASP and NIST, end up with something an auditor, a board and an attacker read the same way. The rogue AI incidents documented since 2016 are what that capability exists to prevent.
Frequently Asked Questions
What is AI red teaming? AI red teaming is structured adversarial testing that tries to make a deployed AI system produce harmful, leaked or unauthorized behavior. Testers work against the running application, including its prompts, retrieval sources and connected tools, rather than the model alone. The output is a set of reproducible failures, each with evidence, a severity rating and a named fix owner.
How is AI red teaming different from penetration testing? A penetration test hunts a defect that either exists or does not, and one verification closes it. An AI red team test hunts behavior that appears in some attempts and vanishes in others, so results are reported as success rates across repeated trials. Scope, tooling, tester skills and retest cadence all differ between the two.
What are the most common AI red teaming attacks? The 2025 OWASP Top 10 for LLM Applications names prompt injection, sensitive information disclosure, supply chain weaknesses, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. For agents, OWASP separately names tool misuse, goal hijacking, memory poisoning and privilege abuse.
Which frameworks are used for AI red teaming? Most programs anchor to four references. OWASP supplies the vulnerability taxonomy through its LLM Top 10 and its GenAI Red Teaming Guide. NIST AI RMF 1.0 supplies the Govern, Map, Measure and Manage structure. MITRE ATLAS catalogs adversary tactics against AI systems, and the EU AI Act sets a legal floor for some model providers.
How often should a company red team its AI systems? Treat it as continuous rather than annual. Run a full engagement before general availability, then retest automatically whenever the model version changes, a new tool is exposed to the agent, a new data source joins the retrieval index, or a new connector goes live. Continuous monitoring covers the gaps between those gates.
Can AI red teaming be automated? Partly. Automation covers more of the risk surface by expanding confirmed attacks across variants, languages and repeated trials, which turns anecdotes into measurable rates. Human testers still find the failures that depend on domain knowledge, policy judgment and business context. Microsoft’s published lessons keep both inside the same program rather than choosing one.
What tools are used for LLM red teaming? PyRIT, Garak, Inspect and Promptfoo appear most often on enterprise shortlists. PyRIT is Microsoft’s MIT-licensed Python Risk Identification Tool for generative AI, and Microsoft also ships an AI Red Teaming Agent inside Azure AI Foundry. Judge any tool on attack coverage, agentic support, scorer quality and reproducibility across repeated runs.
How do you measure whether an AI red team assessment worked? Four numbers make the work reportable. Attack success rate per attack class and model version at a stated trial count, risk-surface coverage against applicable OWASP categories and ATLAS techniques, regression rate for previously fixed attacks after a change, and median time from confirmed finding to verified retest by severity tier.
