Key Insights
The Open Worldwide Application Security Project (OWASP) keeps prompt injection as the top risk in its Top 10 for LLM Applications, published in August 2026. AI red teaming is how a team finds out whether its own system resists that attack, testing it the way an adversary would, using inputs that make it leak data, ignore its rules or take harmful actions.
This guide covers what an AI red team tests, how to run an exercise end to end, how to measure the results and who belongs on the team. Test names follow OWASP and MITRE ATLAS, so findings line up with the catalogs security and engineering teams already use.
What Is AI Red Teaming?
AI red teaming is structured adversarial testing of models, copilots and agents for security and safety failures. Traditional red teams, the origin of the split between red teamer and blue teamer roles, emulate an adversary against an enterprise's whole security posture. The difference lies in the goal, since conventional red teams test whether an attacker can gain access to a system while AI red teams test whether an attacker can get a system to act against its own constraints.

Scope covers benign users as well as attackers, because OWASP notes prompt injection can be unintentional, which keeps the work close to real AI security vulnerabilities.
Why Do AI Agents Change Red Teaming?
Agents call tools, chain actions, maintain memory and access data, so tests must cover the actions and permissions behind each output rather than the output alone. Prompt injection can let an attacker hijack a tool-using agent to run code or exfiltrate data, and an agent that treats an unauthorized request as sufficient authority to act becomes a privileged proxy, the classic confused-deputy problem.

Agent testing also has to reach retrieved content, persistent context and excessive agency, each of which model testing misses:
Retrieved Content: Instructions hidden in an email or document can enter an assistant's retrieval context and lead it to send the user's data to attacker-controlled URLs, sometimes without the victim ever opening the message.
Persistent Context: Memory and context poisoning extends the risk to summaries, embeddings and retrieval stores that carry corrupted data into later runs.
Excessive Agency: For agents built on large language models (LLMs), OWASP traces excessive agency in LLMs to excessive functionality, permissions and autonomy.
Probing those three separately tells an agent's owner whether to remove a tool, narrow a credential scope or add an approval gate.
What Should an AI Red Team Test?
An AI red team tests for prompt injection attacks, sensitive data disclosure, jailbreaks, excessive agency and privilege escalation, pairing a model test with an agent test for each, because the same risk calls for a different test once the system can act on its own output:
Risk | How to test a model | How to test an agent |
Prompt injection (direct and indirect) | Attempt instruction overrides in chat and score whether system rules hold | Plant instructions in documents, emails and tool outputs, then check whether a tool call follows |
Sensitive data disclosure | Probe for training data, system prompt contents and other users' data | Seed canary records, then watch outputs, logs and outbound tool parameters |
Jailbreaks and policy bypass | Run reformulation, role-play and multi-turn escalation per policy category | Check whether a jailbroken session can invoke forbidden tools |
Excessive agency and tool misuse | Test refusal of out-of-scope action requests | Attempt out-of-scope and chained calls and unapproved writes, and confirm approval gates fire |
Privilege escalation through agent credentials | Test whether credentials in context can be revealed | Request reads of config files holding secrets, and attempt actions the user has no rights to |
The agent column covers the sends, writes and credential reads that follow a bad output, which is why testers need staging tools and synthetic data.
How to Run an AI Red Team Exercise
Running an exercise means fixing the scope and the number of runs per case before testing begins, because one cycle's results mean little unless they can be compared with the last.
Define Scope, Rules of Engagement and Success Criteria
Scope names every model, agent, tool and environment under test, plus what is off limits. Testing runs in a controlled environment, with sandboxes standing in for production systems so you can observe agent behavior before release.
Rules of engagement set allowed techniques, data handling and stop conditions, so a policy might permit only synthetic records and test accounts, forbid messages to real external recipients and halt testing at a production write path. Success criteria are fixed before testing starts, as a maximum acceptable attack success rate per category and a rule that any credential read or unapproved write fails the release.
Build a Threat Model from the System's Data and Permissions
The threat model starts from data and permissions rather than prompts, listing the sources the system reads, the tools it can call, the credentials each tool uses and the trust boundaries where untrusted content enters context.
A security team assessing an agent that summarizes inbound vendor email, reads a customer relationship management (CRM) system, and can send email traces the chain from hidden instructions in a message to a CRM lookup to exfiltration through the send tool. Each hop is a place a control can break the chain. For the security operations center (SOC), the same map shows which logs have to exist before an incident can be reconstructed.
Design Test Cases from OWASP and MITRE ATLAS
OWASP's 2026 list for LLM applications keeps prompt injection and sensitive information disclosure at the top and raises excessive agency, while OWASP's 2026 agentic list adds agent goal hijack, tool misuse, identity and privilege abuse, and memory and context poisoning. MITRE's ATLAS knowledge base contributes agent tool invocation, credential harvesting and jailbreak techniques. Each case runs as both a malicious user and a benign one sending ambiguous requests, with carriers built from prompt injection examples.
Run Each Test Repeatedly
Single runs mislead because model outputs vary between attempts. In its January 2025 technical blog, Strengthening AI Agent Hijacking Evaluations, the National Institute of Standards and Technology (NIST) reported taking five injection tasks and attempting each attack 25 times, with the average attack success rate rising from 57% to 80% across repeated attempts.
Each case's model and defense versions, settings and run count are recorded so later results compare like with like. Multi-turn variants escalate across a conversation, and human testers add the adaptive attacks that fixed suites tend to miss.
Triage, Fix and Retest
Testers score each finding by impact (the data reached or the action triggered) and by likelihood (how often it succeeded). Each finding then goes to the application team, the platform team or the model provider.
Fixes work best at the system level, since designers have to assume prompt injection is possible. Common fixes include minimal tool permissions, human confirmation for agent actions, and filtering untrusted content. Closure means rerunning the original case at the run count recorded at scoping, then adding it to a regression suite.
Feed Findings into Monitoring
Each successful attack chain gives the SOC a pattern to detect in production. Teams can turn that pattern into runtime rules such as alerts on tool calls outside an agent's normal pattern, writes to new external destinations, or reads of configuration files holding credentials.
Telemetry logging and continuous monitoring with rollback to known-good behavior both depend on analysts getting the exact tool-call sequence behind each finding. AI Agent Security from Abnormal, currently in private preview, is designed to discover agents across AI platforms, track their privileges and tool access, and detect off-task runs.
How Do You Measure AI Red Teaming Results?
AI red teaming results are measured as rates and trends across repeated runs, reported with uncertainty as standard errors or confidence intervals so real change stands out from noise. Each metric tracks a different way a program slips:
Attack Success Rate by Category: Successful attempts over total attempts per category, reported with run count and spread.
Time to Fix: Days from a confirmed finding to a verifying retest.
Regressions After Updates: Closed findings that reopen after an update.
Agent Tool Coverage: The share of an agent's tools and permission scopes tested.
Read together, regressions beside attack success rate show whether fixes last, and a release that changes tool policies, approval logic or credential scopes without updated tests should not ship.
Who Should Be on an AI Red Team?
AI red teams work best with security engineers, machine learning (ML) engineers and domain experts. NIST's generative AI profile states that "the quality of AI red-teaming outputs is related to the background and expertise of the AI red team itself." Security engineers chain attacks across identity and permission systems, ML engineers explain why a defense failed in retrieval or decoding, and domain experts recognize harmful output in deployment context.
An internal team suits continuous testing because it can rerun cases on every change. Outside specialists fit pre-launch reviews and high-risk agents, where independence counts more than speed.
Treat AI Red Teaming as a Continuous Program
AI red teaming earns its place when one cycle's results change what the next one tests. Because no current mitigation covers every attacker technique, each model, prompt, tool or permission change reopens what the last test showed.
AI Agent Security, currently in private preview, is part of Abnormal's AI Security suite and is designed to track each agent's privileges and tool access and support automated response and containment. Explore AI Agent Security to see how it fits a red teaming program.
Frequently Asked Questions
How is AI red teaming different from penetration testing?
Penetration testing tries to circumvent the security features of a specific application, system or network. To cyber professionals, red teaming often means exactly that, while to AI professionals it usually means testing whether a model's guardrails can be bypassed.
How often should you red team an AI system?
Continuously, with a full retest on every model, prompt, tool or permission change and a lighter regression run on a fixed schedule in between.
Can AI red teaming be automated?
Yes, partly, since automated frameworks pair an attacker model, a target and a judge, and some refine candidate jailbreaks without human intervention, which scales repeated runs and regression suites. Because automated judges can misjudge harm, programs add human experts who find adaptive, context-specific attacks.
Do you need to red team third-party AI models?
Yes, before adoption, because a provider's published benchmarks reflect its own tests rather than the deployer's prompts, integrations, tools and agent permissions, and fine-tuning can change the behavior those benchmarks measured.
What is indirect prompt injection?
Indirect prompt injection reaches a model through content it ingests rather than through what the user types, with an attacker planting instructions in a web page, document or email the model later reads. A third party mounts the attack, and the primary user is often the one harmed.