Key Insights
Stanford HAI's 2026 AI Index Report records a sharp rise in reported AI incidents, reaching 362 in 2025. AI governance failures are the harm that comes from how an organization approved, operated or oversaw an AI system. Each documented case here, from a tribunal ruling against an airline to a government report with fabricated citations, traces back to a gap that was already open before launch.
This guide pairs each failure with the control that would have caught it, then sets out the patterns that connect them and the published controls that address each one.
Why Do AI Governance Failures Happen?
AI governance failures happen when an organization puts an AI system into use before anyone owns its output, tests it on the real task, reviews its actions, limits its data or vets its supplier. None of those gaps needs new technology to close, only an owner assigned before launch, which is the work of the discipline of AI governance.
Which Real AI Failures Show What Weak Governance Costs?
Weak governance has cost organizations a tribunal award, a repaid government fee, a deleted production database and exposed applicant data. A control applied where each failure began could have reduced the risk of that outcome.

Air Canada: Chatbot Gave a Customer Wrong Refund Policy
After a grandparent's death, Jake Moffatt asked Air Canada's website chatbot about bereavement fares and was told he could claim a reduced fare retroactively, which the airline's policy did not allow. In Moffatt v. Air Canada, decided February 14, 2024, British Columbia's Civil Resolution Tribunal called "a remarkable submission" the airline's argument that the chatbot was effectively a separate legal entity responsible for its own actions.
It found Air Canada responsible for all the information on its website and held that the airline had not taken reasonable care to make the chatbot accurate. The tribunal ordered Air Canada to pay C$812.02 in damages, interest and fees.
No one owned the chatbot's answers, so they drifted from the published policy without anyone noticing. Giving each customer-facing AI system a named content owner fixes that, provided the owner runs a regression set of policy questions before release and again after every policy change.
New York City: MyCity Chatbot Gave Incorrect Legal Guidance
On March 29, 2024, The Markup reported that New York City's MyCity business chatbot, launched in fall 2023, was giving answers that contradicted the law:
Bosses could take a cut of workers' tips.
Landlords did not have to accept Section 8 vouchers.
Businesses could refuse cash.
City or state law says otherwise on all three points, and each was the kind of answer a business owner would act on.
What was missing was pre-launch testing on the regulated topics the bot would face, such as housing, wages and payments. A domain test set for each, approved by a program owner or lawyer before launch, supplies that check.
Replit: AI Coding Agent Deleted a Production Database
According to The Register's July 2025 report, SaaStr founder Jason Lemkin said that Replit's AI coding agent had ignored his instructions to freeze code and deleted his production database. He concluded that such agents are "powerful tools with specific constraints, not replacements for understanding what commercial software requires." Replit's chief executive, Amjad Masad, called the deletion "unacceptable and should never be possible" and said the company had started rolling out automatic separation of development and production databases.
Permission design was the weak point, because the agent could write to production and only a prompt instruction held it back. Least privilege paired with human approval before any destructive action addresses it, with authorization enforced in the downstream system rather than left to the model, which is the core control against excessive agency in AI agents.
McHire: Default Admin Credentials Exposed Applicant Data
In June 2025, security researchers found that the administration interface of McHire, the hiring platform built on Paradox.ai's Olivia chatbot that most McDonald's franchisees use, accepted 123456 as both username and password for a test restaurant account. Combined with a flaw in an internal API, that login opened applicant chats and contact details, as the researchers' disclosure shows. The credentials stopped working within two hours of the report, and Paradox.ai confirmed the next day that the issues were resolved.

Vendor review was the missing control. A third-party assessment standard for every AI vendor covers this, as long as it checks credential hygiene, leftover test accounts, and includes the right to evaluate the supplier's processes in the contract.
Samsung: Employees Pasted Source Code Into ChatGPT
Less than three weeks after Samsung lifted a ban on staff using ChatGPT in March 2023, employees in its device solutions and semiconductor business had copied source code from a database download program into the service, uploaded code for identifying defective equipment and uploaded records of a meeting to generate minutes, according to The Register's April 2023 report. Samsung responded by limiting how much each question could upload and warned that ChatGPT could be blocked on the company network if it happened again.
The missing piece was a data rule, since staff were allowed to use the service before anyone had said what could go into it. A data classification rule for AI prompts, paired with a list of approved tools, is the first defense against sensitive information disclosure through AI.
Deloitte Australia: Government Report Contained AI-Generated Errors
Australia's Department of Employment and Workplace Relations commissioned Deloitte to review the Targeted Compliance Framework and published the report in July 2025. The following month, University of Sydney researcher Chris Rudge flagged fabricated references, including a made-up quote from a Federal Court judgment, and a revised version dated September 26 disclosed that an AI tool chain had been used.
In October 2025, the department confirmed that Deloitte would repay the final installment of its contract, The Guardian reported. Deloitte also conceded that it had not followed its review and oversight processes.
Human verification and disclosure both failed. A sign-off step in which a named reviewer verifies every citation and quote addresses the first, and stating where AI was used addresses the second.
Microsoft 365 Copilot: EchoLeak Zero-Click Flaw Exposed Data
In June 2025, Microsoft disclosed EchoLeak, a critical flaw in Microsoft 365 Copilot in which a single crafted email could plant instructions that Copilot followed later, when a user asked an ordinary question. Copilot's answer then sent the user's data to an attacker without the user clicking anything. According to an academic case study of the flaw, Microsoft had already deployed a server-side fix that needed no customer action.
The flaw came down to untrusted input sharing a context with trusted company data. Prompt injection testing modeled on real prompt injection examples surfaces it, and segregating external content while limiting what data each assistant can reach keeps a successful injection from reaching anything worth taking.
What Patterns Connect These Failures?
Missing ownership, untested tasks, absent human review, loose data rules and unvetted vendors connect these failures, and several cases show two of them together. Mapping each pattern to its cases shows which control would have addressed it:
Failure pattern | Cases that show it | Control that addresses it |
No accountable owner | Air Canada, New York City MyCity | Named owner per AI system, recorded in an inventory |
No task-specific testing | New York City MyCity, Microsoft 365 Copilot | Domain accuracy tests and adversarial prompt injection tests before release |
No human review | Replit, Deloitte Australia | Approval gates for destructive actions and sign-off on deliverables |
No data rules or scoping | Samsung, Microsoft 365 Copilot | Prompt data classification, approved tools and scoped data access |
Unvetted vendors | McHire | Third-party assessment with contractual evaluation rights |
Microsoft 365 Copilot appears in two rows because EchoLeak combined untested input handling with an assistant that could read the user's emails and files.
Which Controls Prevent the Most Common AI Governance Failures?
An owner, pre-launch testing, human approval of high-impact actions, data rules, vendor review and deactivation authority all appear in the National Institute of Standards and Technology (NIST) AI Risk Management Framework, ISO/IEC 42001 and guidance from the Open Worldwide Application Security Project (OWASP). Each works only when a named team owns it, as AI governance roles and responsibilities show.
An AI system inventory only helps once you find shadow tools and add them. Abnormal's AI Governance is designed to discover approved and shadow AI tools, attribute ownership and track compliance gaps against frameworks.
Each control gate is a point where a case went wrong:
Pre-Launch Testing: Each system is tested on the real task before deployment and regularly while in operation.
Human Approval: A person approves high-impact actions before taking them.
Data and Vendor Rules: Written rules cover what data may enter AI tools and what each AI vendor must show before onboarding.
Deactivation Authority: A named person can take a system offline when it behaves inconsistently with its intended use.
Every one of these controls was missing in at least one case in this guide.
AI Failures Are Governance Gaps Teams Can Close
Every case in this guide points to a governance control that a named team could apply before launch. Does each AI system have an owner named in the inventory, a dated test record, an approval gate, a data rule and a vendor review on file? A team that can answer yes for every system in use has closed the gaps behind each failure in this guide.
Abnormal's AI Governance, part of the AI Security suite, is designed to discover sanctioned and shadow AI tools and attribute ownership. Explore AI Governance to see how it fits a governance program.
Frequently Asked Questions
What is an AI governance failure?
An AI governance failure is harm from a system nobody had placed under control, as distinct from a model that simply performed badly. A wrong answer in testing is a quality problem, while the same answer reaching a customer, a court or a regulator with no owner, test or reviewer in between is a governance failure.
What is the most common cause of AI failures?
No single failure point dominated the cases here, and most showed two at once. That matters more than any ranking, because a single control rarely would have prevented an incident alone, and programs that close one gap while leaving its usual partner open tend to fail in the same place twice.
Can a company be held liable for what its AI says?
Yes, at least one tribunal already has. A deploying company was held responsible for what its chatbot told a customer, because it was responsible for everything on its own website and had not taken reasonable care to keep the bot accurate.
Who is accountable when AI makes a mistake?
Accountability sits with the organization that deployed the system, not the vendor that built it or the team that prompted it. In practice, that means a named owner inside the deploying organization, with the authority to change the system or take it offline. A vendor contract can share the cost of a failure, but the deploying organization still answers to its customers and regulators.