Points clés
Deepfake attacks are one front in a wider shift: AI has collapsed the barrier between intent and capability in cybercrime, democratizing cybercrime in ways security teams are only beginning to understand. What used to require technical sophistication, cultural knowledge, and significant resources now requires only motivation. Lone scammers, ransomware crews, and nation-state operators all use generative AI to scale their operations. Defenders need to understand the full spectrum to protect the people and payment processes those attacks target.
This article draws on insights from the webinar "The Adversary's New Assistant: Weaponizing AI Chatbots".
Key Takeaways
Synthetic voices and faces exploit the trust people place in what they see and hear, which makes them harder to question than a suspicious email.
Fraud targets wire transfers and executive impersonation, and a single fake video call cost Arup's Hong Kong office $25 million.
Entry-level AI phishing kits and malicious chatbots cost less than $100 per month, putting polished attacks within reach of anyone with intent.
Legacy detection often struggles with indicators AI-generated attacks don't contain, so behavioral baselines carry the load in email, paired with verification controls for voice and video.
What Are Deepfake Attacks?
Deepfake attacks use AI-generated synthetic media (audio, video, or images) to impersonate real people, and they have moved from theoretical concern to operational reality. Deepfakes exploit the human tendency to trust what we see and hear, and that trust tends to be hard to override with security training alone.
For example, a convincing video call with a "CFO" or a voicemail from a "trusted vendor" can bypass the skepticism users apply to emails. In enterprise settings, deepfake attacks tend to follow a few recurring patterns:
Synthetic Executive Requests: A cloned voice or synthetic video of a CEO or CFO pushes an employee toward an urgent wire transfer or a credential handover.
Email-Primed Confirmation Calls: A business email compromise (BEC) message, a form of AI-powered social engineering, sets up the request, and a follow-up call or video meeting with a synthetic executive appears to confirm it.
Hiring and Identity Verification Fraud:Synthetic video and forged documents carry an applicant through remote interviews or know-your-customer (KYC) checks.
Each pattern starts with a message on a channel the organization already trusts. The Hong Kong case covered later in this article began with a message before any synthetic media appeared. Our roundup of AI-powered cyber attack examples covers more real-life deepfake attacks that you can learn from.
How Deepfake Attacks Fuel Cybercrime Democratization
The biggest change AI has brought to cybercrime is accessibility. Sophisticated attacks once required years of technical training, language fluency, and operational infrastructure, and AI has sharply lowered that barrier.
As Piotr Wojtyla, Head of Threat Intel and Platform at Abnormal, explained during the webinar: "As long as you want to do something now, you're enabled to do it. Which pretty much puts a lot of people who previously were not in the place to really carry out attacks. Now as long as they have the intent, they have the capability to do so as well."
Basic attackers who once lacked the skills to write a convincing phishing email can now produce polished messages in minutes. Criminal marketplaces sell "Social Engineering as a Service," with entry-level AI phishing kits and malicious chatbots less than $100 per month. Tools like FraudGPT and WormGPT, built specifically for malicious purposes, remove the guardrails that legitimate AI platforms attempt to enforce.
At the more sophisticated end, online crime groups are operationalizing AI across their entire workflow. Leaked communications from the Black Basta ransomware group show operators using AI for troubleshooting malware, rewriting code, and scaling their operations. Nation-state actors, including Iranian, Russian, and North Korean groups, use large language models (LLMs) for target reconnaissance, parsing stolen data, and developing AI-powered social engineering campaigns.
The scale and speed of this shift show up clearly in the numbers. The FBI's 2025 Internet Crime Report logged 22,364 complaints with a likely AI nexus and $893.3 million in adjusted losses, with BEC, the channel that often sets up a synthetic call, accounting for $3.05 billion. Defenders are already feeling it too. 41% of CISOs in Gartner's 2026 survey reported at least one social engineering incident involving a deepfake in the past 12 months. The same release notes that "AI is increasing the volume, personalization, and credibility of social engineering while reducing the reliability of familiar detection cues."
Together, these findings explain why deepfakes matter beyond the headline cases: the economics now favor attackers at every tier, and the cues that defenders trained users to spot are quietly becoming unreliable.
AI-Generated Content Attacks
The most immediate impact of generative AI on attack quality is the loss of traditional detection signals. Grammar errors, awkward phrasing, and cultural missteps, the tells that security training taught users to spot, are becoming less reliable.
LLMs produce fluent, contextually appropriate text in many languages. A single AI system can generate thousands of unique, personalized phishing messages per hour, and each one can reference specific job responsibilities, current projects, local events, or recent business activities.
But attackers use AI for more than better emails. They're exploiting legitimate platforms to bypass security controls entirely. Attackers have weaponized Gamma AI and Canva, tools designed for presentations and design, to host phishing content. An email arrives from the legitimate platform inviting the recipient to view a shared document. Because it originates from a trusted service, it passes through secure email gateways (SEGs) without triggering alerts. The actual phishing lure lives inside the document, one click removed from the inbox where users have been trained to be vigilant.
In one multi-stage Gamma phishing campaign, the hosted presentation led to a CAPTCHA-gated page and then to a fake Microsoft SharePoint login. An adversary-in-the-middle framework then captured session cookies to get past multifactor authentication.
The pattern across these campaigns is the same: attackers borrow the credibility of services employees already use, so the warning signs land on surfaces no one is trained to inspect. Users scrutinize emails but rarely apply the same suspicion to documents hosted on platforms they trust, or to a routine-looking CAPTCHA gate.
Real-Life Deepfake Attack Examples
Beyond text, AI has enabled identity attacks that were previously impossible. Our overview of AI cybersecurity threats covers the wider range of these attacks.
The most striking documented case involved the Hong Kong office of engineering firm Arup, which lost more than $25 million after a finance employee was convinced to transfer funds during what appeared to be a video call with the company's CFO. The approach began with a message purportedly from the CFO requesting a confidential transaction. On the call itself, every participant other than the victim was a synthetic recreation of a real colleague. The employee processed multiple wire transfers, and the fraud surfaced only when they later checked with head office. Arup later said that "none of our internal systems were compromised."
Voice cloning attacks are becoming similarly sophisticated. In the UK, Starling Bank launched a Safe Phrases verification campaign asking customers to agree a verification phrase with family and friends, and it warns that criminals can clone a voice from as little as three seconds of audio. In July 2024, a Ferrari executive stopped a deepfake impersonation of the CEO by asking the caller a verification question the impostor could not answer.
North Korean operatives have used AI to infiltrate Western companies through job applications. Cultural gaps and unfamiliarity with casual conversation, questions like "What do you do on weekends?", would previously expose them during interviews. AI can help operatives create convincing false identities, complete technical assessments, and perform work after hiring.
The scheme has moved into synthetic video in the interview itself: the FBI now advises employers to ask remote candidates to wave a hand in front of their face on camera, a movement that may prompt AI-generated video to malfunction, and to require unobscured backgrounds.
Inma Martinez, AI scientist and global chair for GenAI and Agentic AI projects at GPAI, framed the broader implication: "Not very capable bad actors were able to do much. And thanks to these tools, you have North Korean hackers applying for IT jobs in North America."
How to Defend Against Deepfake Attacks and Other AI-Enabled Threats
Defending against deepfake attacks requires accepting that detection methods built for older phishing were never designed for synthetic media or AI-written text. The threat spectrum includes opportunistic scammers and nation-state operations alike, and attackers at every tier now use AI tools.
Detect the Email Components of Deepfake Attacks
Legacy email security relies on known indicators of compromise: malicious attachments, suspicious URLs, blacklisted senders. AI-generated attacks contain none of these signals. They are text-based, they originate from legitimate platforms or compromised accounts, and they are crafted to appear normal within existing communication patterns. Signature-based tools often struggle to judge whether a grammatically perfect request from a real vendor domain is out of character. The only viable detection approach for email-borne AI attacks is behavioral AI.
Behavioral AI threat detection approaches the problem from the other direction. By learning what "known good" looks like for an organization, including how employees communicate, how vendors typically interact, when payment requests normally arrive, and what legitimate approvals contain, it makes it possible to surface anomalies even when the specific technique has never been seen before. A banking-detail change sent from a vendor's real domain at an unusual hour, or a "confidential transaction" request that skips the usual approval chain, can stand out against that baseline however polished the text is.
Abnormal applies this model across cloud email and, where deployed, collaboration tools such as Teams and Slack, and it extends existing email security frameworks. Our overview of behavioral AI in email security covers specific applications.
The era of static rules and signature-based detection is over. What replaces it must understand human behavior deeply enough to protect humans from the attacks that now exploit them.
Verify Voice and Video Requests Outside the Inbox
Behavioral AI can help detect the email and account-based components of these campaigns, but a live deepfake video call happens on a channel the inbox does not see. Organizations can pair email defenses with verification controls for voice and videoconferencing:
Out-of-Band Callbacks: Verifying payment or credential requests through a separately known phone number or an in-person check, rather than contact details in the request, can stop a synthetic caller.
Pre-Agreed Verification Phrases: Executive and finance teams can adopt the Safe Phrases model so a caller must prove identity with information a scraped conference recording cannot supply.
Interview Hygiene for Remote Hiring: Hiring teams can apply the FBI's guidance on hand movements and unobscured backgrounds and capture images for comparison with later meetings.
Approval Paths That Do Not Bend to Urgency: An instruction to keep a transfer confidential or skip a second approver is a signal to pause the payment.
This isn't optional. Organizations that hesitate to deploy AI-powered defense are creating an ever-widening gap between attacker capability and defensive posture. The tools available to threat actors will only become more powerful and more accessible. Matching that evolution requires AI-native email security that learns and adapts as quickly as the threats do, paired with verification habits for the calls the inbox never sees. Our guide to email security trends tracks how voice and video deepfakes are increasingly tied to email-borne attacks.
Meet Regulatory and Compliance Obligations for Deepfake Attacks
Regulators and agencies now set obligations and guidance relevant to deepfakes:
EU AI Act Article 50: The Article 50 transparency obligations for synthetic content took effect on August 2, 2026, with a December 2, 2026 deadline for machine-readable marking on systems already on the market.
FinCEN Alert:FinCEN alert FIN-2024-Alert004 asks US financial institutions to use the key term "FIN-2024-DEEPFAKEFRAUD" in suspicious activity reports.
NSA, FBI and CISA Guidance: The agencies' joint guidance, Contextualizing Deepfake Threats to Organizations, recommends real-time verification capabilities and rehearsed response plans.
Finance and compliance teams can map verification controls to these requirements and keep audit-ready records of each callback, approval, and escalation.
Build a Defense That Matches the Pace of AI-Enabled Attacks
Deepfake fraud, AI-written phishing, and synthetic identity schemes share a common thread: they strip away the signals legacy tools were built to catch. Readers should walk away ready to invert the detection question, from "does this look bad?" to "does this look wrong for us?", and to pair that email-side defense with verification habits for the calls and video meetings the inbox never sees.
Abnormal is designed to close the email side of that gap. By modeling the workflows, vendor interactions, and approval patterns that define normal for each organization, Behavioral AI surfaces the socially engineered requests that pass every signature check, and it extends across cloud email and connected collaboration tools such as Teams and Slack. Abnormal was named a Customers' Choice in the 2026 Gartner® Peer Insights™ Voice of the Customer for Email Security, the third consecutive year it has earned the distinction.
See how behavioral AI helps detect deepfake attacks and related threats across the attack spectrum. Request a demo to learn how Abnormal helps protect your organization.
Frequently Asked Questions
How Does Behavioral AI Differ From Traditional Email Security in Detecting AI-Generated Phishing Attacks?
Behavioral AI inverts the detection logic legacy tools rely on. Instead of hunting for malicious artifacts that AI-generated phishing no longer contains, it learns what normal communication looks like for each user, vendor, and workflow, then flags deviations. Traditional security asks whether an email looks bad; behavioral AI asks whether it looks wrong for this organization.
What Specific Steps Should Organizations Take to Verify Executive Requests Made via Video Calls to Prevent Deepfake Fraud?
Pair every sensitive request with a verification step synthetic media cannot replicate. Use callback verification through directory-stored numbers, pre-shared verification phrases between executives and finance teams, and physical gestures such as a hand wave that AI-generated video may struggle to render. Hold approval workflows firm under claims of urgency or confidentiality, and document each step to maintain an audit trail.
How Can Companies Protect Themselves From North Korean Operatives Using AI-Generated Identities to Infiltrate Through Remote Hiring?
Interview practices that exploit current AI limitations are the most practical defense. Require live hand movements and unobscured backgrounds on camera, probe for conversational fluency in unscripted exchanges, and capture frame comparisons across sessions to detect switching between personas. Pair these checks with behavioral monitoring of new hires' access patterns and data movement in the weeks after onboarding.
What Are the Compliance Requirements Organizations Must Meet Under the EU AI Act and FinCEN Guidance Regarding Deepfake Threats?
EU AI Act Article 50 requires machine-readable marking for synthetic content, with existing products needing compliance by December 2, 2026. US financial institutions filing suspicious activity reports tied to deepfake fraud must include the identifier "FIN-2024-DEEPFAKEFRAUD", and NSA, FBI, and CISA joint guidance recommends real-time verification capabilities and documented response procedures.
How Effective Are Pre-Agreed Verification Phrases and Out-of-Band Callbacks at Stopping Deepfake-Enabled Social Engineering Attacks?
Both controls work because they rely on information attackers cannot obtain from scraped audio, video, or intercepted communications. A private verification phrase creates a shared secret outside any model's training data, and a callback to a verified number forces attackers to abandon the attempt or expose themselves. Effectiveness depends on consistent enforcement, especially when requests are framed as urgent or confidential to discourage verification.