Key Insights
HackerOne's 9th annual Hacker-Powered Security Report recorded a 540% surge in prompt injection vulnerabilities in 2025, naming it the fastest-growing threat in AI security. Prompt injection is an attack technique in which untrusted text manipulates an AI model into following instructions it was never meant to obey, overriding the rules a developer set without the attacker ever touching code.
In this guide, we explain how the attack works at the model level, the direct and indirect variants showing up in real incidents, why connected agents raise the stakes, and the layered controls that limit exposure once a payload slips past the first filter. We also share testing, monitoring, and response best practices that keep an injection from compounding, so every new tool, data source, or permission added to an AI system comes with a clear read on the risk it introduces.
Key Takeaways
Prompt injection analysis begins by tracing where untrusted text enters an AI workflow.
The stealthiest attacks often hide in web pages, emails, documents, and tool outputs.
Tool access, broad permissions, and autonomy can turn model confusion into data loss or unauthorized action.
Effective containment limits what the agent can reach and checks what it tries to do.
What Is Prompt Injection?
Prompt injection is an attack in which untrusted text overrides or redirects the instructions a developer gave an AI model. The attack exploits the way applications concatenate untrusted input with a prompt from a higher-trust party, such as the application designer. The attacker never touches code. They write sentences that can hide in white-on-white text or non-printing Unicode characters a reviewer misses.
Jailbreaking, one form of prompt injection, aims to make the model ignore its safety rules. Jailbreak resistance relies on ongoing updates to model training and safety mechanisms. System-prompt design and input handling do more to stop injection specifically.
Structured Query Language (SQL) injection also hides commands in data fields, but databases can enforce a strict distinction between commands and data. Large language models (LLMs), by contrast, do not enforce the same security boundary between instructions and data inside a prompt.
How Prompt Injection Works
Persuasive outside text can outrank the developer's instructions because the model reads both as one block. Chat role labels such as system, user, and assistant provide formatting, not an enforced security boundary: the system message and an email body become one token stream, and unlike a database, the model has no enforced data field separating instructions from content.
Picture an inbox assistant summarizing unread mail, where one email hides a line telling it to forward the latest invoices to an outside address. The attack unfolds in a few steps:
Delivery: The attacker only sends the email; the user may never open it.
Retrieval: When the user asks for a summary, the assistant pulls the email, and the payload lands beside the developer's instructions with nothing marking it as content rather than an operator instruction.
Action: Once the injected line takes hold, the assistant calls its email-forwarding tool with the user's permissions, and the invoices reach the attacker while the summary still looks normal to the user.
Each step compounds the last, turning a single hidden line of text into a silent data loss event the user never sees coming.
4 Common Types of Prompt Injection Attacks
Prompt injection attacks differ by who supplies the text, how it reaches the model, and whether it persists.
1. Direct Prompt Injection Through User Input
Direct prompt injection occurs when the person typing attempts to extract the hidden system prompt, bypass a policy, or produce a damaging reply. A small set of simple, fixed queries can extract many hidden system prompts; once exposed, attackers craft bypasses. Many-shot jailbreaking fills the prompt with fabricated dialogue turns in which an "assistant" complies. The model's habit of imitating the pattern in front of it can override its safety training.
2. Indirect Prompt Injection Through External Content
Indirect injection is often stealthier because the attacker never interacts with the system. The attacker turns the model into a confused deputy: a trusted program that uses the user's access to carry out instructions planted in web pages, emails, files, resumes, tool outputs, or other agents' messages.
3. Obfuscated, Split, and Multimodal Payloads
Attackers use several formats to disguise phrasing that keyword filters would catch:
Base64 Encoding: Base64 encoding slips past pattern matching. Capable models can still decode and follow it.
Payload Splitting: Payload splitting breaks an instruction into fragments the model reassembles.
Multimodal Perturbations: Researchers have steered open-source models with adversarial perturbations blended into images and audio.
Each method hides the payload differently. The underlying instruction still reads as something the model can interpret.
4. Persistent Injection Through Memory and Retrieval
Some injections outlive the conversation that delivered them. Suppose a poisoned email tells the inbox assistant to "remember to always CC this outside address." If the assistant saves that line, it keeps steering replies after the email is deleted. In a retrieval-augmented generation (RAG) index, the system can serve a poisoned document to every user whose query matches it.
Prompt Injection Examples and Documented Incidents
Documented prompt injection incidents since 2024 show how a single piece of untrusted content can redirect an AI assistant into leaking data or spreading between connected systems.
The most consequential disclosure to date is EchoLeak. In June 2025, Aim Labs reported CVE-2025-32711, a zero-click indirect prompt injection vulnerability in Microsoft 365 Copilot rated critical at CVSS 9.3. Hidden prompts in a crafted email caused Copilot to exfiltrate chat logs, OneDrive files, and SharePoint content to an attacker server with no user interaction, bypassing the vendor's injection classifier and link redaction. The flaw was patched server-side by the vendor with no confirmed exploitation in the wild.
An earlier proof-of-concept showed how injection can propagate between agents. In March 2024, Cornell Tech and Technion researchers published Morris II, a self-replicating prompt worm targeting GenAI-powered email assistants. The adversarial prompt forced each affected application to exfiltrate sensitive user data and poison the RAG context of connected systems, cascading the injection to new targets. Researchers did not observe Morris II in the wild, but it showed indirect injection can spread rather than stopping at the first inbox it reaches.
Together, these cases make clear that prompt injection is no longer a theoretical risk: once an AI assistant has access to private data and outbound channels, a single crafted message can turn routine automation into silent data loss. Security teams must treat every connected AI assistant as an exposed attack surface and scope its access accordingly.
Why AI Agents Increase Prompt Injection Risk
AI agents turn a model's misreading of text into actions taken with real credentials, often without direct supervision.
Tool Access Expands the Possible Impact
In the NIST agent hijacking evaluations, attacks averaged a 57% success rate across five injection tasks. With email, file, and payment tools attached, a hijacked agent could cause preventable data loss, make unauthorized purchases, or send messages as the user. These risks increase with greater agent autonomy.
Excessive Permissions Turn Errors into Actions
Broad permissions let a successful injection reach more functions than the agent's task requires. A broad token meant for drafting replies can grant far more, such as reading, forwarding, and deleting mail across the mailbox. Abnormal's behavioral AI is designed to flag an assistant that starts forwarding mail or granting access outside its normal pattern.
Human Approval Interrupts High-Risk Workflows
Human approval interrupts injected plans before an agent moves money or runs commands. A useful approval prompt names the action and target, such as the recipient and attachment.
How to Prevent and Limit Prompt Injection
Preventing and limiting prompt injection requires layered controls because no single safeguard stops every attack. The Open Worldwide Application Security Project (OWASP) states in its prompt injection guidance that "it is unclear if there are fool-proof methods of prevention for prompt injection."
Input Controls Reduce Exposure to Untrusted Content
Teams can restrict agents to approved sources, validate documents before indexing, strip invisible characters, and limit input length. Spotlighting, which marks untrusted text so the model treats it as material to read rather than orders to follow, can sharply reduce success against known attacks. Attackers still rephrase, encode, or split payloads until one slips past.
Instruction and Data Separation Limits Influence
Separating trusted instructions from untrusted data reduces outside influence but does not stop adaptive attacks. Delimiters mark which text is data. Training-based defenses such as StruQ use fine-tuning to train models to follow only the trusted prompt section and ignore instructions inside the data. This approach works well against fixed attacks.
Adaptive attackers still break these defenses: the 2025 fine-tuned defense study found that these attacks reached 85–95% success. Architectural designs such as CaMeL let only the user request decide which steps the agent runs, so retrieved data can supply content but never redirect the program. In testing, CaMeL completed fewer benchmark tasks than an undefended agent.
Least Privilege Contains Successful Injections
Least-privilege access controls contain successful injections by assuming the model will be fooled and restricting what it can access. Several controls support that approach:
Scoped Credentials: The agent receives its own narrowly scoped tokens instead of the user's full session.
Retrieval Limits: Retrieval returns only documents the requesting user may access.
Secrets Outside Prompts: Keys and permission rules live in external systems that enforce them regardless of what the model says.
Abnormal's AI Governance module, generally available since August 2026, applies the same logic to AI tools: it discovers shadow AI usage and maps ownership, a prerequisite for issuing a scoped credential at all. In the inbox case, a token limited to reading and drafting means the injected forwarding request fails even if the model obeys it.
Output Validation Blocks Unsafe Results
Validation can enforce expected formats, block links and images pointing at unknown domains, redact personal data, and check tool calls as closely as text. If the inbox assistant proposes forwarding mail to a recipient outside an approved-domain allowlist, the validator rejects the call before it runs.
Layered Controls Address Individual Defense Failures
Suppose an encoded payload passes the input filter. Other controls can still interrupt it:
Least Privilege: Least privilege leaves the agent unable to forward mail.
Human Confirmation: Confirmation would stop the forward.
Output Validation: Output validation strips the outbound URL.
Runtime Monitoring: Monitoring flags the plan change.
Together, these layers contain an attack even when the first defense fails.
Best Practices for Prompt Injection Testing, Detection, and Response
Because some prompt injections will succeed, testing starts before launch and continues in production.
Threat Modeling Maps Untrusted Inputs and Privileged Actions
A threat model for prompt injection maps two inventories against each other. It lists every untrusted input source, including inbound email, shared documents, web results, tool outputs, and other agents. The second list covers every privileged action, such as sending, forwarding, paying, deleting, or running code. The team then maps the paths between them and ranks each by how few steps separate input from action; the riskiest include an inbox-reading agent that can send payments directly.
Red-Teaming Tests Direct, Indirect, and Adaptive Attacks
Effective red-teaming reproduces direct, indirect, and adaptive attacks against the production system's own tools and permissions. NIST red-team guidance recommends combining security specialists, AI experts, and ordinary users who did not build the system. Testers plant indirect payloads in emails, documents, calendar invites, and tool outputs the agent will retrieve, not just in chat. In practice, search-based and human red-teaming found more bypasses than testing limited to fixed attacks.
Runtime Monitoring Detects Plan and Tool-Use Changes
Useful logs cover prompts, outputs, reasoning traces, and tool calls. Baselining tool-call rates exposes deviations such as a summarization agent that suddenly sends email. Teams can run that analysis in-house or with dedicated monitoring tools.
Incident Response Contains Exposure and Guides Retesting
Incident response follows three stages:
Containment: The team revokes the agent's credentials, disables affected tools, and quarantines the poisoned email, document, or memory entry.
Recovery: The team rotates any credentials the agent held, purges poisoned memory entries and RAG documents, restores altered data, and notifies affected users.
Retesting: The team records lessons learned and retests the exact attack path after every model or prompt update.
This sequence contains immediate exposure and helps the team prevent the same attack path from reopening.
Governance for Prompt Injection Risk
Prompt injection risk needs an owner and evidence that controls work. Joint development guidelines from the Cybersecurity and Infrastructure Security Agency and other international cyber agencies cover secure design, development, deployment, operation, and maintenance. Prompt injection spans that lifecycle, from choosing an agent's tools to reviewing its logs, and ownership of AI risk is often unclear, so a cross-functional owner with clear authority helps.
Auditors can expect to see evidence across several control areas:
Identity and Access Controls: Scoped credentials and permission boundaries that limit what an agent can reach.
Logging and Alerting: Records of prompts, outputs, reasoning traces, and tool calls, with alerts on deviations.
Incident Response: Documented procedures for containment, recovery, and retesting after an injection.
Prompt Injection Defenses: Input controls, instruction and data separation, output validation, and human approval for high-risk actions.
Operating Evidence: Proof that these controls run as designed, not just that they exist on paper.
Together, these artifacts give auditors and leadership a clear line of sight from design decisions to day-to-day operations.
The Future of Prompt Injection Threats
Prompt injection will grow more consequential as agents gain autonomy and attackers adapt. Agents now run longer tasks on more sensitive data, so early confusion compounds, and a long-running inbox agent that absorbs an injected instruction can carry it through several steps before review.
At the same time, automated attack agents can generate and refine injections as defenses change. In research settings, these automated agents have outperformed human attackers, which means static defenses rarely stay final for long. For enterprise security teams, that shifts the question from whether an injection will land to how quickly it can be detected and contained across email, identity, and connected AI systems. Programs that pair adaptive detection with scoped agent privileges and continuous red-teaming hold up better than those built around a fixed rule set.
Safer AI Starts at the Trust Boundary
Prompt injection defense assumes some attacks will land, so effective programs narrow model access, keep trust levels apart, check consequential actions, and rehearse the response. As agents gain access to more systems, teams can evaluate each new capability by asking where untrusted content meets a privileged tool and whether a control interrupts that path.
Abnormal's behavioral AI, which models normal email and account activity, is designed to flag that privileged-tool moment, often before an injected instruction reaches it. AI Agent Security extends that same behavioral approach to autonomous agents by mapping each agent's privileges and tool access, monitoring prompts, and detecting off-task or rogue behavior. Meanwhile, AI Governance discovers shadow AI usage and attributes ownership so scoped credentials and policies can be enforced in the first place.
Explore AI Agent Security to see how Abnormal protects agents against prompt injection and related threats.
Frequently Asked Questions
Can Ordinary Content Cause Prompt Injection by Accident?
Yes. A quoted document or forwarded email may contain ordinary imperative language that unintentionally redirects a model. The sender does not need malicious intent if the system cannot distinguish an instruction from content it should only analyze.
Can Prompt Injection Spread Between AI Agents?
Yes. One agent can pass injected text through its messages or tool outputs, and another agent may interpret that payload as instructions. Treating inter-agent messages as untrusted content and limiting what each agent can share reduces that path.