TL;DR
Generative AI security is the set of controls that protect copilots, chatbots, RAG apps, and AI agents from misuse, data leaks, and unsafe actions. In practice, it covers seven layers, from the prompt a user types to the tools an agent can call. The biggest real-world risks are prompt injection, sensitive data exposure, over-permissioned agents, and unapproved shadow AI. For this reason, each risk needs a preventive control and a detective control, owned by a named team. Finally, OWASP’s Top 10 for LLM Applications and NIST’s Generative AI Profile give a solid starting checklist. Start with an AI inventory and access controls, then add guardrails, monitoring, and red teaming in phases.
Key Takeaways Generative AI security protects seven layers, including prompts, models, retrieval data, agents, outputs, supply chain, and shadow AI. IBM found 97% of organizations with an AI-related breach lacked proper AI access controls. Every threat needs both a preventive control and a detective control with a named owner. In particular, RAG systems must enforce the same document permissions users already have, at retrieval time. AI agents need their own identities, least-privilege scopes, and human approval for high-impact actions. A phased roadmap beats a big-bang program, starting with inventory and access before advanced testing. Where the Real Exposure Sits IBM’s 2025 Cost of a Data Breach research puts numbers on the generative AI security gap. It found that 97% of organizations that suffered an AI-related security incident lacked proper AI access controls. The same research also found that 63% had no AI governance policies to manage AI or prevent shadow AI.
Watch on YouTube
Can LLM Gateways Make Enterprise AI Safer?
How an LLM gateway centralizes model access, logging, and policy checks so every GenAI app follows the same security rules.
Those numbers describe a gap in basics. Most enterprises bought GenAI tools faster than they extended identity, data, and monitoring controls to cover them.
So closing that gap takes a map of where GenAI systems can be attacked, and a clear control for each point. This guide lays out that map, layer by layer, and turns it into a working plan.
What Is Generative AI Security? Generative AI security is the practice of protecting systems that generate text, code, images, or actions from attack, misuse, and data exposure. It covers the model, the data the model can reach, the tools it can call, and the people and systems that use its output.
The scope has grown quickly. Early projects were single chatbots, while today’s deployments include RAG assistants over company documents and agents that file tickets, send emails, or change records.
Generative AI Security vs Traditional Application Security Traditional application security assumes structured inputs and predictable code paths. By contrast, a generative AI system accepts free-form language and decides at run time what to say or do, which changes where controls must sit.
Table 1: How generative AI security differs from traditional application security
Area Traditional application security Generative AI security Inputs Structured fields validated against rules Free-form language, files, and web content that can carry hidden instructions Logic Deterministic code paths Probabilistic model behavior that changes with context Data access Queries defined by developers Retrieval and tools chosen by the model at run time Identity Human users and service accounts Humans plus agents acting on their behalf Testing Code scans and penetration tests Adversarial prompt testing and red teaming, repeated after each change
The Generative AI Attack Surface in Seven Layers Security teams get more value from a layered map than from a flat list of risks. Each layer has different owners, different controls, and different ways to fail.
Layer 1. Users and Prompts This is where people type requests and paste content. Risks include direct prompt injection, jailbreak attempts, and staff pasting confidential data into a tool that stores it.
Layer 2. The Model The model can be manipulated, extracted, or tricked into revealing its system prompt. In addition, fine-tuned models add the risk of poisoned training data that changes behavior in subtle ways.
Layer 3. Retrieval Data and Vector Stores RAG systems pull company documents into the model’s context when a user asks a question. If retrieval ignores document permissions, the assistant can show a junior employee a board memo, and poisoned documents can plant instructions for the model to follow.
Layer 4. Agents and Tools Agents turn text into actions such as API calls, database writes, and emails. An agent with broad permissions turns a successful prompt injection into a real business incident, which is why this layer carries the highest impact.
Layer 5. Outputs Model output then flows into screens, documents, code, and other systems. Unchecked output can leak data, carry malicious code into downstream apps, or spread confident but wrong information.
Layer 6. Model and Component Supply Chain Third-party models, open-source weights, plugins, and libraries all enter the stack from outside. So each one needs the same vetting as any other software dependency.
Layer 7. Shadow AI Unapproved AI tools used by staff sit outside every control above, since security never reviewed them. IBM’s research found that high levels of shadow AI added USD 670,000 to the average breach cost, and our guide to shadow AI covers how to find and govern it.
OWASP Top 10 for LLM Applications 2025, Mapped to Layers The OWASP Top 10 for LLM Applications 2025 is the most widely used catalog of LLM application risks. Mapping each item to a layer also tells you which team owns the fix.
Table 2: OWASP LLM Top 10 (2025) mapped to layers and first controls
OWASP item Main layer First control to put in place LLM01 Prompt Injection Prompts, retrieval data Separate trusted instructions from untrusted content and limit what the model can do LLM02 Sensitive Information Disclosure Retrieval data, outputs Permission-aware retrieval plus output DLP LLM03 Supply Chain Supply chain Approved model and component registry with vetting LLM04 Data and Model Poisoning Model, retrieval data Controlled data sources and change review for training and index content LLM05 Improper Output Handling Outputs Treat model output as untrusted input in every downstream system LLM06 Excessive Agency Agents and tools Least-privilege tool scopes and human approval for high-impact actions LLM07 System Prompt Leakage Model Keep secrets and access rules out of system prompts LLM08 Vector and Embedding Weaknesses Retrieval data Access-controlled vector stores and tenant separation LLM09 Misinformation Outputs Grounding with citations and human review for high-stakes answers LLM10 Unbounded Consumption Prompts, model Rate limits, token quotas, and cost alerts per user and app
Two items deserve extra attention in 2026. Excessive agency grows as agents gain tools, while vector weaknesses grow as more company knowledge moves into RAG indexes.
Kanerika Service
AI Governance Services
Kanerika designs AI governance programs that cover inventory, policy, data protection, and monitoring for enterprise GenAI and agent deployments.
Explore AI Governance Prompt Injection in Practice, and Why Filters Alone Fail Prompt injection sits at the top of the OWASP list because it is hard to stop at the source. Language models cannot reliably tell the difference between instructions from their owner and instructions hidden inside the content they read.
Direct Injection A user types instructions meant to override the system prompt, such as asking the assistant to ignore its rules or reveal its configuration. Input screening catches many known patterns, but attackers rephrase faster than filter lists update.
Indirect Injection The more dangerous form arrives through content the model processes on the user’s behalf. For example, a shared document, a web page, or an email can contain hidden text that tells an assistant to forward data or call a tool.
Picture an agent asked to summarize a supplier’s invoice. A line in white text tells it to email the last ten invoices to an outside address, and an agent with open email access may do exactly that.
Layered Defense That Holds Limit the blast radius , since an agent that cannot send external email cannot leak data by email, whatever the prompt says.Isolate untrusted content by marking retrieved documents and web pages as data, so they stay separate from system instructions.Screen inputs and retrieved content , but treat injection classifiers as one signal and never as the only control.Gate sensitive actions so any action that sends data out or changes records needs confirmation.Log and alert whenever a tool call does not match the user’s original request.The first control does the most work. Filters reduce how often attacks succeed, while tight permissions cap how much damage a successful attack can cause.
The Generative AI Security Control Matrix A risk list does not tell anyone what to build. A control matrix does, because it pairs each threat with a preventive control, a detective control, and an owner.
Table 3: Generative AI security control matrix
Threat Preventive control Detective control Typical owner Prompt injection Input screening, instruction isolation, narrow tool scopes Prompt and response logging with injection pattern alerts AI platform team Sensitive data leakage Data classification, permission-aware retrieval, masking Output DLP and sensitive-label alerts Data governance teamAgent misuse Agent identities, least privilege, approval gates Tool-call audit trail and anomaly detection Identity and security teams Unsafe output Output filters and encoding before downstream use Sampled human review and user flagging Application owner Supply chain compromise Approved model registry and vendor assessment Dependency and model version scanning Security architecture Cost and resource abuse Rate limits and token budgets Usage and spend anomaly alerts FinOps and platform team Shadow AI Approved tool catalog and clear usage policy Network and SaaS discovery for AI tools CISO office
Owners matter as much as controls. A control with no named owner decays within a quarter, because no one reviews its alerts or updates its rules.
Data Protection Controls for GenAI Apps Most real GenAI incidents are data incidents. The model reveals something it should not have been able to see in the first place.
Classify Before You Connect Label data as public, internal, confidential, or regulated before any AI system can reach it. The data classification best practices guide explains how to make labels stick across large estates.
Enforce Permissions at Retrieval Time Before it answers, a RAG assistant must check that it only retrieves documents the asking user can already open. Filter by the user’s identity at query time, because the model cannot reliably hide content it has already received.
Mask and Redact Sensitive Fields Remove personal data and secrets before content enters prompts, indexes, or logs. Automated redaction agents such as Kanerika’s Susan handle this step for documents at scale.
Watch What Leaves Apply data loss prevention to model outputs as well as inputs. The guide to Microsoft Purview data loss prevention shows how existing DLP policies can extend to AI interactions.
Controls by Deployment Pattern Not every GenAI app carries the same risk. A drafting assistant with no data access needs far lighter controls than an agent that can move money, so match the control depth to the pattern.
Table 5: Priority controls by generative AI deployment pattern
Pattern Main risk Priority controls Public chat tools used by staff Confidential data pasted into external services Approved tool list, enterprise plans with data retention controls, browser and network DLP Enterprise copilots Oversharing through existing file permissions Permission cleanup before rollout, sensitivity labels, usage monitoring Custom RAG assistants Retrieval of documents the user should not see, poisoned content Permission-aware retrieval, source vetting, output DLP, citation grounding Task agents with tools Unauthorized actions after prompt injection Agent identities, least-privilege scopes, approval gates, tool-call audit Customer-facing chatbots Jailbreaks, harmful or wrong answers, cost abuse Input and output guardrails, topic limits, rate limits, escalation to humans
Enterprise Copilots Need a Permissions Cleanup First A copilot that searches company files respects the permissions those files already have, even though many of those permissions are too broad. Years of broad sharing links and open team sites mean it can surface content that people could technically open but never knew existed.
Fix oversharing before rollout rather than after the first complaint. The guide to data access governance covers how to find and close overly broad access at scale.
AI Assessment
Where Does Your AI Program Stand?
Assess readiness across data, governance, security, and operations, and get a clear view of the gaps to close before scaling GenAI.
Start Your AI Assessment → Customer-Facing Bots Carry Brand Risk Too A public chatbot that gives a wrong refund policy or an offensive answer creates legal and reputational exposure, even without any data leak. Keep public bots on narrow topics, ground them in approved content, and hand off to people when confidence is low.
Securing the Model and Component Supply Chain Every GenAI app depends on parts built elsewhere. Foundation models, open-source weights, embedding models, vector databases, plugins, and orchestration libraries all arrive from outside the organization.
Keep an AI bill of materials so every model, version, dataset, and library each app uses is on record.Use an approved model registry so production use is limited to models that passed security and legal review.Verify sources and integrity before deployment by downloading weights from trusted publishers and checking hashes.Review vendor terms before signing, and confirm how each provider stores, retains, and uses prompts and outputs.Scan dependencies continuously , since AI libraries carry known vulnerabilities like any other code dependency.The supply chain also includes the protocols that connect agents to tools. Standards such as the Model Context Protocol make integration easier, and each connected server still needs the same vetting as any third-party component.
Identity and Access for AI Agents Agents are the newest identity type in most enterprises. Treat them as accountable actors, with their own credentials and limits, instead of letting them borrow a human’s full access.
Give each agent its own identity so every action traces back to one agent and one task.Scope tools narrowly by granting read access by default and adding write access only when a named action needs it.Require approval for high-impact actions such as payments, external emails, record deletion, and permission changes.Use short-lived credentials , because tokens that expire quickly limit damage when a session is hijacked.Log every tool call , including the agent, the request, the tool, and the result, so audit and incident response have a full record.Agent risk grows with autonomy. The guides to agentic AI risks and rogue AI governance go deeper on accountability for autonomous agents.
Guardrails and Output Controls Guardrails are policy checks that run before and after the model. They reduce risk, but none of them is perfect, so they work best in combination with access limits.
Input Guardrails Screen prompts and retrieved content for injection patterns, restricted topics, and sensitive data. Also keep untrusted content clearly separated from system instructions.
Model-Level Controls Use system prompts for behavior guidance only, since secrets or access rules placed there can leak. Route requests through a gateway that enforces approved models and logging, as covered in our look at the Databricks Unity AI Gateway .
Output Guardrails Check responses for sensitive data, policy violations, and unsafe code before they reach users or systems. Ground high-stakes answers in cited sources, which also reduces the LLM hallucination problem.
Monitoring and Incident Response for LLM Applications A GenAI app without logs cannot be investigated. So monitoring has to capture enough context to reconstruct what the model saw and did.
What to Log User identity, application, and session Prompts and responses, with sensitive fields masked Documents retrieved and their sensitivity labels Tool calls, parameters, and results Model and prompt template versions A Five-Step Response Playbook Detect problems when alerts fire on injection patterns, data-label hits, or unusual tool calls.Contain the incident by disabling the affected agent, tool, or data source while other AI apps keep running.Investigate by replaying logs, so the team can find the entry point and what data or actions were involved.Remediate by fixing permissions, filters, or content, and notify affected parties when required.Improve the test suite by adding the attack, so the same path is checked on every release.Case Study
AI Compliance Agent With Human Review Built In
A compliance vetting agent that gathers cited findings and maps them to criteria, so reviewers clear backlogs faster while staying in control of every decision.
Read the Case Study → Red Teaming and Continuous Testing Standard penetration tests miss attacks written in plain language. Instead, GenAI systems need adversarial testing that tries to make the model misbehave through prompts, documents, and tool use.
Test prompt injection, jailbreaks, data extraction, agent misuse, and harmful output. Repeat the tests after every model, prompt, or data change, because behavior can shift with each update.
For buyers comparing providers and methods, the AI red teaming buyer’s guide covers scope and evaluation. A one-time AI security assessment is a useful baseline before continuous testing begins.
Governance Frameworks and Regulations That Apply Frameworks give structure to controls and help prove due care to auditors and regulators. Four are most relevant to generative AI security programs today.
NIST AI Risk Management Framework has, since January 2023, organized AI risk work into govern, map, measure, and manage functions.NIST AI 600-1 is also called the Generative AI Profile . The agency released it on July 26, 2024, adding GenAI-specific risks and actions to the AI RMF.EU AI Act obligations for general-purpose AI models started to apply in August 2025, while further obligations phase in over the following years.ISO/IEC 42001 , the AI management system standard , sets requirements for governing AI across an organization and can also be certified.In practice, framework choice depends on geography, industry, and audit needs. The comparison in our AI security framework guide helps teams pick a primary framework and map the others to it.
A Phased Roadmap for Generative AI Security Trying to build every control at once stalls programs. Instead, a phased roadmap delivers risk reduction early and builds toward continuous operations.
Table 4: Four-phase generative AI security roadmap
Phase Focus Key outputs 1. Foundation Visibility and policy AI inventory, usage policy, approved tool catalog, data classification 2. Secure build Controls in new apps Permission-aware retrieval, agent identities, guardrails, AI gateway 3. Production operations Monitoring and response Logging, alerting, incident playbooks, cost controls 4. Continuous assurance Testing and governance Red teaming cadence, framework mapping, metrics reviews
Metrics That Show Whether It Works Security leaders need numbers that show progress to the board. Track a small set that covers coverage, risk, and response, rather than dozens of vanity metrics.
Inventory coverage is the share of AI apps and agents that are registered and reviewed.Access hygiene is the share of agents that have their own identity and least-privilege scopes.Test pass rate is the share of red-team attack cases blocked on the latest release.Data incidents count sensitive-data hits in outputs, which should trend down over time.Response time measures the time from alert to containment when an AI incident occurs.Shadow AI trend tracks how many unapproved AI tools are detected each month.Common Generative AI Security Mistakes The same mistakes repeat across early programs. Each one is cheap to avoid when caught early.
Treating a GenAI app like any other web app , since standard scans miss prompt-level and agent-level risks.Letting agents inherit human permissions , because one injected instruction then carries the full reach of that user.Indexing data before classifying it , so confidential files end up in retrieval results for the wrong people.Relying on system prompts for security , even though instructions in prompts can leak or be overridden.Ignoring shadow AI , when the biggest exposure often sits in tools security never approved.Testing once at launch , although behavior changes with every model and data update, so tests must repeat.The wider list of business-level concerns, from bias to IP exposure, is covered in our guide to generative AI risks .
How Kanerika Helps Secure Generative AI Securing GenAI works best when security is designed into the data and AI platform from the start. That is the approach Kanerika uses to build and govern enterprise AI systems, backed by ISO 27001, ISO 27701, and SOC 2 certifications.
The Delivery Approach Assess AI apps, agents, data sources, and shadow AI, then map them to the seven layers.Design identity, retrieval permissions, guardrails, and logging for each use case before build starts.Build and deliver generative AI and RAG development with controls built in rather than added later.Govern by applying classification, DLP, and access policies through kanGuard and kanGovern services delivered on Microsoft Purview, as well as AI governance programs.Operate the program, then keep monitoring, incident playbooks, and a regular testing cadence running.Case Study. A Compliance AI Agent Built With Review Built In A client that vets outside experts faced growing compliance backlogs, with slow manual research across news, social media, and professional records. In response, the Kanerika team built an AI compliance agent that gathers findings, cites sources, and maps them to disqualification criteria.
At the same time, the design kept people in charge of decisions. The agent handles research while the compliance team reviews cited findings, which cleared the backlog and raised throughput without adding headcount.
What Kanerika Teams Watch For Across AI deployments, the same weak points recur. Retrieval that ignores document permissions, agents with write access they never use, and logging that captures prompts but not tool calls are the most common findings.
Fixing those three before scaling closes most of the practical exposure. It also makes later audits against NIST or ISO 42001 far easier.
Talk to Kanerika
Securing a GenAI Rollout?
Talk to Kanerika’s AI team about mapping your copilots, RAG apps, and agents to the right controls before they scale.
Book a Meeting → Wrapping Up Generative AI security is a layered discipline. Prompts, models, retrieval data, agents, outputs, supply chain, and shadow AI each need their own controls and owners, so no single tool covers the whole problem.
Start with visibility and access, because most breaches exploit missing basics. Then add guardrails, monitoring, and testing in phases, and measure progress with a short set of metrics.
Teams that treat security as part of AI design ship faster, because they spend less time fixing incidents after launch.
Frequently Asked Questions
What is generative AI security? Generative AI security is the set of controls that protect copilots, chatbots, RAG assistants, and AI agents from attack, misuse, and data exposure. It covers prompts, models, retrieval data, agent tools, outputs, third-party components, and unapproved shadow AI, with preventive and detective controls owned by named teams across the organization.
What are the biggest generative AI security risks? The biggest risks are prompt injection, sensitive data disclosure, over-permissioned agents, insecure handling of model output, supply chain weaknesses, and shadow AI. The OWASP Top 10 for LLM Applications 2025 catalogs these risks. In practice, most incidents trace back to missing access controls and weak data governance rather than exotic attacks.
How is LLM security different from traditional cybersecurity? LLM security deals with free-form language inputs and probabilistic behavior instead of fixed code paths. Attacks can hide in documents or web pages the model reads, and agents can take actions based on that content. Traditional controls still apply, but they need AI-specific additions such as guardrails, retrieval permissions, and adversarial testing.
How do you prevent prompt injection attacks? No single filter stops prompt injection, so defenses must be layered. Limit what the model and its agents can do, keep untrusted content separate from instructions, screen inputs and retrieved content, require approval for sensitive actions, and log tool calls for anomalies. Tight permissions cap the damage when an attack succeeds.
How do you secure a RAG application? Secure a RAG application by classifying data before indexing, enforcing the user’s own document permissions at retrieval time, vetting and monitoring the sources that feed the index, and applying data loss prevention to outputs. Keep vector stores access-controlled and separated by tenant, and ground answers in cited sources.
How do you secure AI agents? Give every agent its own identity, grant least-privilege tool scopes, and use short-lived credentials. Require human approval for high-impact actions such as payments, external emails, and record changes. Log every tool call with the agent, request, and result, and test agents regularly against injection and misuse scenarios.
Which framework should enterprises use for generative AI security? Most enterprises combine several. The OWASP Top 10 for LLM Applications guides technical controls, the NIST AI Risk Management Framework and its Generative AI Profile structure risk management, and ISO/IEC 42001 provides a certifiable management system. Organizations serving the EU also need to track EU AI Act obligations.
What is shadow AI and why does it matter? Shadow AI is the use of AI tools that security and IT teams have not approved. It matters because those tools sit outside access, data protection, and logging controls. IBM’s 2025 research found that high levels of shadow AI added USD 670,000 to the average cost of a data breach.
How often should generative AI systems be red teamed? Red team generative AI systems before launch and again after every significant model, prompt, tool, or data change. Many teams also run automated adversarial test suites on each release and schedule deeper manual exercises quarterly. Each confirmed attack path should become a permanent regression test.
How do you measure generative AI security? Track a short set of metrics. Useful ones include the share of AI apps inventoried, the share of agents with their own least-privilege identities, the red-team attack block rate, sensitive-data hits in outputs, time to contain AI incidents, and the number of unapproved AI tools detected each month.
