How to Detect AI-Generated Phishing: Signals and Checks

Knowing how to detect AI generated phishing starts with unlearning the checklist most people were trained on. For two decades, the standard advice was to scan for typos, broken grammar, and awkward phrasing. Large language models removed that tell. The FBI’s Internet Crime Complaint Center warned in a December 2024 public service announcement that “generative AI reduces the time and effort criminals must expend to deceive their targets,” specifically citing corrected grammar and polished phrasing as a way AI-written lures now evade the filters and instincts trained on older, sloppier scams. The tells didn’t disappear — they moved from spelling to structure, and from the message body to the infrastructure behind it.

Why grammar stopped being a signal

Pre-LLM phishing was written fast, often by non-native English speakers working from templates, and it showed: subject-verb disagreement, wrong prepositions, inconsistent tone. Spam filters and human reviewers both leaned on that. A phishing kit that runs its lure text through an LLM before sending strips out nearly all of it. The FBI’s advisory groups this under a broader shift: AI lowers the cost of producing convincing text, images, and even voice and video impersonation at scale, which means the old heuristic — “if it reads well, it’s probably legitimate” — now works against defenders instead of for them.

That doesn’t mean AI-written phishing is undetectable. It means the signal moved from surface errors to statistical patterns in how the text is constructed.

The stylometric signals that still work

Academic research on this problem has converged on a consistent finding: AI-generated phishing has a measurable “voice,” even when it’s grammatically flawless. Synthetic media runs into a comparable ceiling, covered in deepfake detection methods 2026. A stylometric detection study out of Teesside University built 47 features across lexical, syntactic, stylistic, and tonal categories — including imperative-verb density (words like “click,” “verify,” “confirm”), clause complexity, and pronoun patterns that mimic authority or urgency. An XGBoost classifier trained on those features hit 96% accuracy and a 0.99 AUC score distinguishing GPT-4o-generated phishing from legitimate email. The most predictive features weren’t spelling-related at all — they were urgency markers and sentence complexity, patterns that are hard for a generation prompt to fully mask because they’re baked into how instruction-following text gets phrased.

A newer cross-model study published in Frontiers in Big Data tested this against a harder problem: does a detector trained on one LLM’s phishing output still work against a different model’s output? Using a corpus of human-written phishing plus AI phishing generated by GPT-4.1, DeepSeek 3.2, and Llama 3.3 70B, the researchers found within-model detection stayed strong (F1 scores of 0.96–0.97), but cross-model transfer degraded sharply — F1 dropped as much as 28 percentage points when a detector trained on one model’s output was tested against another’s, in the worst case falling to 0.472. The underlying discriminative signal was still there (ROC-AUC stayed above 0.96 in every combination tested), but the decision threshold calibrated for one LLM didn’t transfer. Retuning thresholds with a small sample of the target model’s output closed most of that gap, and training a single detector on a pooled, multi-LLM corpus performed best of all — F1 = 0.997 per generator. Two features held up across every model tested: politeness density and type-token ratio (a measure of vocabulary variety). Practically, that means a detector — or an analyst — trained only against ChatGPT-style phishing may miss lures from a different model family entirely.

What actually still gives it away

Outside of automated stylometry, the reliable manual checks have shifted to context and infrastructure rather than prose quality. The injection-side equivalent, where the payload targets a model rather than a person, is what a prompt injection attack is. The checks:

• Requests that skip a normal process. A wire transfer approval, credential reset, or vendor payment change that bypasses the usual chain of confirmation is a stronger signal than any phrasing quirk.
• Sender domain vs. display name. AI-polished text is often paired with a spoofed or lookalike domain. Check the actual sending address and reply-to header, not the display name.
• Out-of-band verification. The FBI’s advisory specifically recommends establishing a pre-agreed verification phrase for high-trust contacts and independently contacting the purported sender or organization through a known-good channel before acting on a financial or credential request — advice that applies equally to AI-written business email compromise.
• Delivery-mechanism shifts. Attackers are also routing around text-based filters entirely. Microsoft’s Q1 2026 threat landscape report recorded QR-code phishing volume up 146% over the quarter and CAPTCHA-gated phishing more than doubling in March, both techniques designed to move the payload somewhere text-analysis tools can’t read it. A perfectly written email that routes you to a QR code or a CAPTCHA-gated link before showing you the actual phishing page is now a common pattern, not an edge case.

Building layered defenses

No single signal is sufficient on its own, which is why detection research increasingly recommends layering. For organizations building or buying detection tooling, the practical lessons from the current research are specific: don’t trust a classifier trained against a single LLM’s phishing samples to generalize, prefer detectors trained or recalibrated against a mixed corpus, and treat urgency-plus-politeness combinations and unusual clause structure as stronger signals than surface fluency. Teams evaluating guardrail and content-filtering tooling for this problem can find comparative frameworks at guardml.io, and ongoing disclosure tracking for phishing-adjacent AI vulnerabilities and incidents is maintained at ai-alert.org.

One caution on tooling choice: general-purpose “AI text detectors” are a different product from phishing detection, and they do not substitute for it. Published evaluations put their false-positive rates high enough on writing by non-native English speakers to make them unusable as an accusation, and the reasons are set out in how to tell if an email was written by AI. Sender authentication and process verification answer the question those tools only approximate.

Placing those layers inside an existing security program is its own question, because phishing generated by an LLM straddles the email-security controls you already run and the AI-specific catalogs. AI security frameworks compared sets out which of NIST AI RMF, the OWASP LLM Top 10, MITRE ATLAS and ISO/IEC 42001 owns which part of that, and where a program that adopts only one of them has a gap.

On the human side, Microsoft’s report recommends pairing detection with structural controls that make a successful phish less damaging even when it lands: phishing-resistant MFA, passwordless authentication, Safe Links and Safe Attachments scanning, zero-hour auto purge, and automatic attack disruption in endpoint tooling. None of these depend on catching the AI-generated text itself — they assume some percentage of it will get through and limit what a successful click actually buys the attacker.