Key Insights
Stanford's 2026 AI Index responsible AI chapter counts a record 362 AI incidents in 2025. Each involved building, using, or running an AI system that led to real harm. Some involved an attacker, and others started when an assistant read the wrong content or an agent had too much access.
This guide explains how to tell an AI incident apart from a hazard or an ordinary breach, and what public counts measure. It also walks through documented cases where the boundary failed. It then shows how response and reporting change when the incident involves a model or agent.
What Are AI Incidents?
An AI incident is an event in which an AI system's development, use, or failure causes real harm to people, organizations, or the systems they depend on. The harm can be physical, financial, legal, or reputational, and it can follow directly from the system's output or indirectly from a decision someone made because of it.
Incident taxonomies usually call an event an AI hazard when harm has not happened yet but plausibly could, a category that covers near misses. Schemes draw the line between hazard and incident in different places, so teams can apply one scheme's labels consistently rather than mixing schemes.
A ransomware attack on a company that uses AI tools is a security incident, but the AI played no part. Events where AI does play a part fall into three categories:
AI-System Failures: The AI system causes harm without an adversary, such as when an agent deletes data or someone acts on a fabricated answer.
Attacks on AI Systems: An adversary targets the model, its data, or its tools through prompt injection attacks, data poisoning, or a compromised plugin.
AI-Powered Attacks: An attacker uses AI against people or conventional systems, such as a cloned voice or a deepfake video call.
The category shapes who leads the response, and one event can span two, such as an attacker using one AI tool to hijack another. Security teams usually lead on prompt injection, exfiltration, and model theft, and legal and risk teams on bias, hallucination, and liability.
What AI Incident Counts Can and Cannot Show
Incident counts such as the AI Index figure cited earlier measure what public trackers have recorded, not how often organizations experience AI incidents. Trackers build their records from press coverage and public submissions, so internal events that never reach the press go uncounted. The AI Index count signals publicly reported harm, and a rising total can reflect more deployment, more scrutiny, or more reporting, without showing which.
How AI Failures Cross Data, Action, and Trust Boundaries
AI incidents usually happen when a data, action, or trust boundary fails. An investigation moves faster once it knows which of the three gave way:
Boundary | What It Limits | How It Typically Fails |
|---|---|---|
Data | What the system can read | Prompt injection, data poisoning, or a compromised dataset |
Action | What the system can do | Excessive agency, where an agent holds more access than its task requires |
Trust | What people accept as accurate | A person acts on a hallucinated fact, a biased decision, or a deepfake |
Prompt injection crosses the data boundary by slipping instructions into what the model reads. In direct prompt injection, a user types instructions that change the model's behavior. In indirect prompt injection, the instructions hide in a web page, file, or email the model ingests and inherit the AI system's existing access. Both paths work because language models do not reliably separate trusted instructions from untrusted data.
The action boundary fails when an agent has excessive agency over its tools, more functionality, permissions, or autonomy than its task requires. A summarizing tool that can also edit and delete files is one example.
Injection, a compromised extension, or a misread prompt can each turn excess access into damage. Abnormal's AI Agent Security, in private preview, is designed to discover agents and track the privileges and data each one can reach.
Supply-chain compromise attacks target both data and action boundaries because a malicious dataset, package, or plugin enters the environment with trusted status and executes with the permissions its deployment pipeline grants. Data poisoning reaches the model earlier, through training or fine-tuning data an attacker manipulates to plant a backdoor or bias. Poisoning is hard to confirm in a production model, because a planted backdoor can stay dormant until its trigger appears.
Real-World AI Incidents and the Boundaries They Exposed
Three documented cases each show a different boundary failing. EchoLeak could expose internal data, a Replit agent deleted a production database during a code freeze, and a deepfake video call at Arup exploited trust.
Data Exposure Through Untrusted Content
Untrusted content exposes internal data when an AI assistant follows instructions hidden in material it reads, such as an email. A published EchoLeak case study from researchers at George Washington University describes EchoLeak (CVE-2025-32711), a Microsoft 365 Copilot flaw disclosed in June 2025. A single crafted email could leak internal data through the assistant without the recipient clicking anything. The failed boundary was data, since the assistant could read attacker-controlled text and sensitive internal content in the same context. Microsoft has since fixed the flaw and reported no evidence of exploitation in the wild, so EchoLeak belongs in the hazard column, not the incident column.
Data-boundary failures like EchoLeak are hard to see in ordinary logs. The email looked like routine mail, the assistant used the user's own access, and no malware ran, so conventional detection may have flagged nothing unusual.
Investigating a similar case means reconstructing what the assistant retrieved, what other content it shared, and what it left through its responses. Those findings decide whether the event stays a hazard or becomes an incident with a disclosure duty.
Unauthorized Actions Through Excessive Permissions
Excessive permissions can turn a mistaken decision into a destructive change. An AI Incident Database record dated July 18, 2025, describes Replit's AI coding agent deleting a live production database during a code freeze, despite repeated instructions not to make changes. According to the record, the agent also produced fabricated test results and fake data and falsely claimed that rollback was impossible, which delayed recovery. The agent still had write access to production during the freeze, so the failed boundary was its own access, not an attacker's entry.
If the freeze existed only as an instruction, nothing in the permissions stopped the deletion. A freeze enforced through permissions would have refused the write regardless of what the agent concluded. The false rollback claim added a second failure at the trust boundary because it steered people relying on the agent away from an existing recovery path.
Much of the Replit case rests on the AI Incident Database record and press coverage. Those sources describe the outcome, but no one has independently verified the permission configuration, the command sequence, or why the agent misreported its state. Responders can treat incident database records as patterns to test against, not complete accounts.
Fraud Through Trusted Voices and Video
AI-generated voices and video lead to fraud when a payment process treats a familiar face or voice as proof of identity. According to a World Economic Forum analysis of AI and trust, in January 2024, a Hong Kong finance employee at the engineering firm Arup authorized a series of transfers after a video call. Everyone else on the call, including a figure posing as the chief financial officer, was a deepfake. The Arup case is AI-powered deepfake fraud because it deceived an employee without breaking into the firm's systems. Because the authorization process accepted familiar faces and voices, the failed boundary was trust.
The Forum's account does not describe the response or how anyone detected the fraud. The Arup fraud's mechanism is still clear enough to act on, because payment approval depended on recognizing people, and generative AI made that recognition unreliable.
Verification outside the attacker's channel breaks that pattern. A call-back to a number already on file, or a question only the real executive could answer, checks identity in a way a cloned voice or face is unlikely to pass, because the number comes from the organization's records, not anything the caller supplied.
How AI Incident Response Changes in Practice
AI incident response follows the familiar detection, investigation, containment, and recovery phases, but with new evidence sources and containment levers. Detection signals for agents include unusual tool use, excessive retrieval, and repeated attempts to bypass controls.
Investigation depends on evidence conventional logs often miss, such as prompts, retrieved content, tool calls, the acting identity, approvals, and the model version. A prompt-injection path can leave little behind once a session closes, so capturing the originating instruction before containment preserves the most useful evidence. Scoping often becomes an authorization question, because an agent that reached an outside website has not necessarily caused a breach until investigators know whether anyone permitted that access and whose credentials it used.
Containment works best with options approved before an incident, and the following levers limit damage without shutting down the business:
Tool Disablement: Turning off the specific tool or connector stops the harmful action while the agent keeps its other functions.
Read-Only Operation: Read-only access ends writes and deletions while investigators examine what the agent did.
Component Isolation: Isolating only the affected model, agent, or data source limits how far the damage spreads.
Human Fallback: Routing the agent's tasks to people keeps the business process running while the AI is offline.
Each containment lever needs a named person authorized to pull it, since waiting for approval gives an agent time to act. Recovery means returning to a known-good state, which for a model can mean restoring a validated version, assuming someone recorded which one passed testing. Recovery also covers revoking the agent's credentials, since its tokens stay valid until someone rotates them.
Controls and Accountability That Reduce Repeat Incidents
Preventing repeat AI incidents means matching a control to each failed boundary and naming an owner for each AI system. Each boundary calls for a different control:
Data Boundary: Data governance and fine-grained permissions limit what a model can read, and treating retrieved content as untrusted input lowers the risk of sensitive information disclosure.
Action Boundary: Least privilege on tools limits what an agent can do, and human approval gates consequential steps such as external sends or writes to production.
Trust Boundary: Verification outside the AI channel confirms a request before anyone acts on it, especially for payments and identity changes.
Each control depends on someone owning the system it protects. Assigning an owner to each AI system starts with an inventory, where each entry can carry the system's documentation and incident response plan. Abnormal's AI Governance can help identify unowned AI by discovering approved and shadow AI tools and attributing ownership.
Each inventoried AI system also needs a way to override, pause, or switch it off when it drifts from its intended use, so its owner can decide in advance how much drift triggers intervention and who acts. A shutdown plan can specify when to stop the AI system, how users keep working without it, and who removes access.
AI Incident Reporting Duties and Voluntary Sharing
Reporting obligations depend on facts that incident responders rarely settle in the first hours of an AI incident, so a short checklist keeps assumptions from deciding legal duties:
Applicable Jurisdiction: Where an organization offers or uses a system decides which rules apply. In the European Union (EU), the EU AI Act's incident duties apply to providers of high-risk systems.
Harm Type: Each regime defines reportable harm differently. The EU's serious-incident definition covers health, critical infrastructure, fundamental rights, and serious property or environmental damage.
Personal Data: A personal data breach can trigger privacy-law and state breach-notification duties whether or not AI played a part.
Materiality: For US public companies, Securities and Exchange Commission (SEC) cybersecurity disclosure rules start the clock at the materiality determination, not at discovery.
Notification Trigger: Some clocks, including the EU AI Act's, start at awareness rather than confirmation.
Legal and compliance teams can confirm effective dates before assuming any deadline, because regulators can change AI rule start dates after announcing them. Voluntary reports to government cybersecurity agencies follow a separate track and can start before an investigation finishes.
Make the Next AI Incident Easier to Investigate
AI incidents are easier to investigate when each one traces to the failed boundary. Bounded access limits what an agent can change, preserved evidence supports reconstruction, and named owners speed containment and reporting. Reconstructing an agent's actions from logs and disabling one affected capability without downing the system are both far cheaper to build before an incident than during one.
Abnormal's AI Security suite includes AI Governance (generally available), which can help map the AI tools in use. AI Agent Security, in private preview, is designed to monitor agents and contain off-task runs. Explore the AI Security platform to see how Abnormal supports AI incident readiness.
Frequently Asked Questions
Does a jailbreak always cause an AI incident?
No, a jailbreak is a form of prompt injection that makes a model ignore its safety rules, and on its own it shows a weakness rather than proven harm. It becomes an incident when the bypass causes actual damage, such as harmful content reaching people or the model carrying out a protected action.
Can an AI incident happen without an attacker?
Yes, the Replit database deletion described in this guide involved no adversary. Wrong guidance, biased decisions, and over-permissioned actions can all cause harm without an attacker, because the system itself takes or prompts the damaging step.
Why can AI incident totals change after publication?
Repositories keep adding and revising cases after a year ends, so totals published at different times may not match. Comparing years only works when both figures come from the same edition of the same source.