2025 was the year AI security stopped being primarily a research area and became a practitioner problem. The shift was driven by the deployment scale reaching the threshold where operational attacks became financially worthwhile.
This is the review of what mattered.
1. Attacks went operational
The defining shift of 2025: the attack classes documented in research — prompt injection, jailbreaks, model extraction — transitioned from theoretical demonstrations to documented operational incidents.
By mid-2025, incident databases were tracking multiple cases per month of LLM applications being exploited for: unauthorized data access via prompt injection in enterprise tools, fraud facilitation via jailbreaks in customer service bots, and data exfiltration via model memorization exploits.
The incidents weren’t sophisticated. Most involved well-documented attack classes against applications with no injection detection layer. The attackers were opportunistic. The lesson: documented vulnerabilities without deployed defenses become operational incidents when deployment scale is large enough.
2. The supply chain attack surface became real
By 2025, supply-chain attacks targeting ML infrastructure had clearly moved from hypothetical to a recognized, recurring threat class. The representative patterns:
- Malicious or typosquatted PyPI/npm packages aimed at ML engineers (similar-name and install-time-payload attacks)
- Tampered model weights distributed through public model hubs, including look-alike or namespace-confusion repositories
- Credential-exfiltration payloads embedded in ML-adjacent JavaScript/Python packages
The ML supply chain is structurally more exposed than traditional software because the dependencies are larger (model weights are GBs, not KBs), provenance verification is less mature, and the install patterns (developers pulling new models frequently) create more attack surface.
The response from the ecosystem was slow but real. Hugging Face improved organization account security and model signature verification. PyPI implemented new policies for high-impact package namespaces.
3. Regulatory enforcement on AI widened
2025 extended AI-specific enforcement that had started in earlier years:
- FTC orders against companies over deceptive claims about what their AI products could do
- ICO (UK) issued guidance with enforcement implications for LLM data processing under GDPR
- The EU AI Act began applying in stages: the prohibited-practice and AI-literacy rules from 2 February 2025, and the obligations for general-purpose AI models from 2 August 2025. The high-risk system requirements had not started applying; the European Commission’s current timeline puts them at 2 December 2027 for the Annex III categories and 2 August 2028 for AI built into regulated products
- US federal requirements came from budget policy rather than NIST. The NIST AI Risk Management Framework (AI 100-1) is explicitly voluntary; it was OMB Memorandum M-25-21 of 3 April 2025, which replaced M-24-10, that directed agencies to implement minimum risk-management practices for high-impact AI
The enforcement actions were mostly civil penalties and consent decrees rather than criminal proceedings, but the message was consistent: claims about what an AI product can do are subject to truth-in-advertising standards, AI applications processing personal data have GDPR obligations, and AI used in high-risk decisions requires documentation.
4. Model hardening genuinely improved
The honest counterpoint: the major model providers got significantly better at resistance to common attacks.
Persona jailbreaks that worked against GPT-3.5 in 2023 fail reliably against GPT-4 class models in 2026. The DAN jailbreak family is effectively dead against frontier models. Many-shot jailbreaking has been partially mitigated.
The improvement is real but asymmetric: hardening improved most for well-documented attack classes on frontier commercial models. Open-weight models, fine-tuned models, and novel attack classes remain more vulnerable.
5. The detection tooling market matured
2025 produced a recognizable category of AI security tooling. At the end of 2024, the options were mostly research frameworks with production aspirations. By the end of 2025:
- Dedicated injection detection tools such as Lakera had production deployments at scale
- LLM observability platforms (Arize, LangSmith, Helicone) were tracking millions of requests
- Security scanning tools (Garak, PyRIT) had CI/CD integration paths
The tooling isn’t complete — coverage gaps remain significant, particularly for novel attacks and non-English content. But the category exists and is producing differentiated products.
What to watch in 2026
- Multi-modal attacks (vision model injection) are early-stage in 2025; expect development in 2026 — the synthetic-media detection side is surveyed in deepfake detection methods 2026
- AI-assisted social engineering continues to scale; phishing quality will improve, which is why how to detect AI-generated phishing now turns on structure rather than spelling
- Governance consolidates around a small set of reference documents rather than a single winner; AI security frameworks compared maps how NIST AI RMF, MITRE ATLAS, the OWASP LLM Top 10, and ISO/IEC 42001 divide the work
- Regulatory enforcement will expand beyond FTC civil penalties to include sector-specific enforcement in healthcare and finance
- Open-weight model security becomes the next frontier — the commercial model hardening story doesn’t apply to the billions of open-weight model deployments
For dated weekly coverage: aisecweekly.com. For incident tracking: aiincidents.org. To build a 2026 reading feed around the threats above, try the personalized AI-Sec digest builder or browse the topic and resource library.
