Skip to main content

Oct 17, 2026

What Is Dark AI?

Dark AI is attackers' misuse of AI and LLMs for phishing, BEC and deepfake fraud. Explore how it differs from shadow AI and how to defend against it.

Key Insights

A USENIX Security 2026 field study found LLM-based spear phishing almost tripled click rates over generic phishing, at about $0.03 per email.

The FBI's 2025 IC3 report says chat generators can quickly write official-sounding emails that mimic a CEO.

Verizon's 2026 DBIR found most AI-assisted malware development tracked well-known attack techniques.

The UK NCSC expects AI to make intrusions more effective and efficient through 2027 by improving existing tactics.

A USENIX Security 2026 field study found that spear phishing written with large language models (LLMs) almost tripled click rates compared with generic phishing, at about $0.03 per email. Dark AI is the use of artificial intelligence, especially LLMs, by attackers to research targets, write lures, impersonate people and build malicious code. Its toolkit includes criminal chatbots sold on underground forums, mainstream models jailbroken past their safeguards, open-weight models stripped of refusal training, and deepfake voice and video.

For a chief information security officer (CISO) or security operations center (SOC) team, the exposure runs through email and payment workflows. A fluent request from a convincing sender, with no link or attachment to scan, can move money before anyone questions it.

This guide distinguishes dark AI from shadow AI and shows how attackers apply it to business email compromise (BEC), deepfakes, criminal LLMs and reconnaissance. It also covers why those attacks slip past signature-based tools and which controls reduce the risk.

How Operator Intent Defines Dark AI

The model that drafts a customer reply can also draft a CEO impersonation email, and the operator's intent separates the two. Dark AI is misuse in that sense: deliberately working around the restrictions a model's owner places on harmful output, or using a model that never had them.

Most dark AI speeds up attacks defenders already know, such as phishing and malware development. The UK National Cyber Security Centre (NCSC), in its assessment of AI's impact on the cyber threat through 2027, expects AI to make intrusions more effective and efficient by improving existing tactics rather than creating novel threat vectors.

How Dark AI Differs From Shadow AI and Other AI Misuse

Dark AI is defined by hostile intent from outside the organization, while the related terms describe who uses a model and what they aim at. Each term points to a different owner, from fraud prevention to AI governance:

Term

Who Uses It

Intent

Example

Dark AI

External attackers

Fraud and intrusion

Criminal LLM writing BEC lures

Shadow AI

Employees

Faster work outside approved tools

Unapproved chatbot drafting reports

Jailbroken AI

Attackers or users testing limits

Bypass model safeguards

Prompts that override refusals

Uncensored model

Attackers and hobbyists

Remove built-in refusals

Open-weight model with safety training stripped

Adversarial AI

Attackers targeting AI systems

Manipulate or compromise models

Prompt injection, data poisoning

Shadow AI is AI use that isn't captured in an organization's approved systems and processes, and the employee behind it is usually trying to get work done, making data exposure its main risk. Jailbroken and uncensored models often supply dark AI, because a prompt that slips past a model's restrictions, or a downloaded model with its safety training removed, gives an attacker the same fluency without the refusals. Adversarial AI is different again, since it treats the AI system itself as the target.

How Attackers Use Dark AI

Dark AI lowers the skill, time and cost needed to run convincing attacks, and its business impact shows up in inboxes and payment approvals. Each technique applies that advantage at a different stage of an attack.

Four stages of a dark AI attack: reconnaissance, social engineering, malware development and evasion

AI-Written Phishing and Business Email Compromise

LLMs remove the effort that once limited spear phishing to high-value targets. The FBI's 2025 IC3 Annual Report notes that chat generators can quickly create official-sounding emails mimicking a company's CEO, carrying phishing links or directions to wire funds, while adding that not every BEC tactic relies on AI. In the USENIX field study, human-crafted spear phishing still scored highest but doesn't scale, and LLMs close that gap at a fraction of the cost.

Some variants carry no payload at all, and a request to update a vendor's bank details gives a scanner nothing to detonate. Attackers use AI to match the tone and phrasing of the person they impersonate.

Deepfake Voice and Video Impersonation

Deepfakes carry BEC from the inbox into live calls, as a January 2024 case at engineering firm Arup showed. A finance employee in Arup's Hong Kong office received a message that appeared to come from the UK-based chief financial officer about a secret transaction.

Four steps of the Arup deepfake video call that ended in transfers to attacker accounts

According to the South China Morning Post's report, the employee then joined a video conference in which, Hong Kong police said, the people on screen were computer-generated copies of company staff. The employee then made a series of transfers to accounts the attackers controlled.

Nothing in that chain required the employee to ignore a warning, because the video call looked like the verification step. A mandatory callback to the chief financial officer on a directory number before release is the control built for deepfake-assisted payment requests, since it moves verification to a channel the attacker does not control.

Malicious LLMs Sold on Criminal Forums

Malicious LLMs are real products, though many are thin wrappers of uneven quality. FraudGPT was advertised in 2023 as a subscription bot for generating scam content. A USENIX Security 2024 study from Indiana University Bloomington examined these services, including WormGPT and FraudGPT, and found they ran on uncensored models or on public LLM APIs accessed through jailbreak prompts, usually sold by subscription.

The same study found that quality varies widely, with some services rarely producing compilable malicious code or readable phishing emails. Attackers also abuse hosted models and APIs directly, a shift traced in the rise of malicious AI in the cloud. A criminal brand name is therefore a poor guide to a tool's capabilities, and the safer assumption is that fluent, tailored lures are cheap for anyone who wants them.

Automated Reconnaissance and Malware Development

AI shortens the research and coding that precede an attack. The NCSC's assessment says threat actors are almost certainly already using AI for reconnaissance, vulnerability research, social engineering and basic malware generation. For email attacks, reconnaissance means mining public profiles, org charts and vendor relationships to build a believable pretext. Verizon's 2026 Data Breach Investigations Report (DBIR) found most AI-assisted malware development tracked well-known attack techniques, and the NCSC expects skilled actors to automate rapid changes to malware and infrastructure to evade detection.

Why Dark AI Attacks Are Harder to Detect

Dark AI strips out the signals that filters and trained employees have relied on to spot fraud. A secure email gateway (SEG) and other signature-based filters were built around malicious links, attachments, known-bad domains and sender reputation.

Signature-based filters often struggle with a text-only BEC message from a compromised vendor account or a newly registered lookalike domain. Abnormal's Inbound Email Security uses behavioral AI to detect text-only BEC, vendor fraud and AI-crafted social engineering by analyzing organization-wide communication patterns, message context and environmental risk signals, with a rationale for each verdict.

Human cues are fading as well, and Verizon's 2026 DBIR, citing the reported increase in AI-assisted phishing emails, moved its guidance away from frequent typos as a detection cue. Voice cloning removes another cue, since a familiar voice on the phone no longer proves who is speaking.

How Organizations Defend Against Dark AI

Defense against dark AI pairs process controls that verify requests with behavior-based detection that judges context rather than content. Behavior-based detection flags deviations such as a vendor bank-detail change requested from an unfamiliar location, and it runs alongside SEGs and native Microsoft 365 or Google Workspace filtering.

Verification works best when it never relies on the channel the request arrived through. Finance teams can confirm transfers and bank-detail changes by calling a number they identify independently, require a second approver for vendor payment changes and send a small test payment before moving larger sums to new account details.

Training works best when it focuses on the request rather than the writing. Fear, urgency and unexpected requests for money, passwords or secrecy are better warning signs than spelling, and teams can agree on code phrases for sensitive approvals. Reporting should also cover calls and texts, where Verizon's 2026 DBIR notes attackers increasingly use voice and mobile-centric techniques and where deepfake attacks land.

Preparing Your Organization for AI-Powered Attacks

Dark AI raises the quality of lures without changing what attackers want: money, credentials and access. Out-of-band verification, request-focused training and behavioral detection address those attacker goals from different angles, and each one still works when the writing is flawless and the voice on the call sounds right.

Explore Abnormal's Inbound Email Security to see how behavioral AI is designed to surface payload-less BEC, vendor fraud and AI-generated phishing alongside native Microsoft 365 and Google Workspace protections, with a rationale for each verdict.

Frequently Asked Questions

Is dark AI the same as shadow AI?

No, dark AI is attacker misuse of AI, while shadow AI is employee use of tools the organization has not approved. Shadow AI usually comes from people trying to work faster, and its main risk is sensitive data landing in unvetted services. Fraud and threat teams typically own dark AI response, while AI governance or IT owns shadow AI policy.

Are malicious LLMs real products?

Yes, criminal sellers market malicious LLMs as subscriptions, and academic researchers have bought and tested them. Many are wrappers that send jailbreak prompts to a mainstream model or run an open-weight model with safeguards removed. Quality varies, and their clearest effect is cheaper access to fluent, tailored text.

How do you detect AI-written phishing?

AI-written phishing is detected through context rather than writing quality. Useful signals include a first-time sender, a vendor changing payment details, an unusual login location behind the sending account and requests that pair urgency with secrecy. For payment requests, a callback to a known number remains the deciding check, because polished writing proves nothing about who sent it.

Can email security tools detect dark AI attacks?

Some can, depending on what they look for, since filters built around malicious links, attachments and known-bad senders see little in a text-only request from a trusted-looking account. Detection that compares each message with how the organization and its vendors normally communicate has more to work with, because the request itself, such as a sudden bank-detail change, is the unusual part.

How do deepfakes lead to payment fraud?

Deepfakes supply the voice or face an employee would normally treat as proof. A typical scheme opens with an email from a supposed executive, then moves to a call where a cloned voice confirms the request and stresses confidentiality. Because the call seems to settle the doubt, the employee approves the transfer, and callbacks and dual approval break the scheme by using channels the attacker does not control.

Protect Against Evolving Email Threats

See how behavioral AI detects attacks that legacy defenses miss.