Field notes on the AI attack surface — shadow LLMs and agents, prompt injection, MCP supply-chain risk, and the regulation that's about to demand you have answers. Written by the team building AI Security Posture Management.
Practical, vendor-neutral writing for security teams putting AI into production — and hardening the APIs, web apps, containers and cloud it all runs on. No hype, no lorem ipsum: every piece is meant to leave you with something you can act on this week.
Unsanctioned LLMs, agents, and MCP servers are already in your fleet — here's why discovery has to come first.
Direct vs. indirect injection, real-world examples, and why OWASP ranks it as the number-one LLM risk.
The adversarial-testing duty, the evidence you'll be asked for, and the timelines that are already live.
Over-broad tool scope, typosquatting, and tool poisoning — plus a hardening checklist you can run today.
The second-ranked LLM risk is also the quietest: models that leak the very data they were trusted to handle.
A tribunal ruled an airline liable for its chatbot's bad advice — and rejected the idea that the bot was a separate legal entity.
Retrieval-augmented generation, the vector database it runs on, and the security seams that open up the moment you bolt your data onto a model.
A decades-old security flaw is having a renaissance — because an AI agent is the most powerful confused deputy we have ever built.
The Model Context Protocol exists to kill the N×M integration problem — and in doing so it turns every tool connection into a shared trust surface.
MCP gives an agent hands. The orchestrator is the nervous system that decides when to use them — which makes it the component that holds the credentials and the blast radius.
An LLM application is not just a model — it is a prompt-assembly pipeline with one critical trust boundary most teams draw in the wrong place.
Perceive, plan, act, observe — the loop that makes an agent an agent is also the mechanism that lets one poisoned observation hijack everything that follows.
Three agent capabilities that are each harmless alone become a data-theft machine the moment one system has all three at once.
An MCP tool description is prose the model obeys — which makes the server that writes it a place to hide instructions your agent will follow.
Vectors are not anonymised numbers — they are a lossy copy of your text, stored in a database that usually has none of your usual access controls.
Everything the model reads — your rules, the user's text, retrieved docs, tool output — lands in one flat buffer with no privilege separation. That is the whole problem.
Every model you ship is assembled from parts you did not build — scraped data, downloaded weights, community adapters and third-party plugins. Each is an injection point.
An agent acting for a user needs authority without becoming the user. That is the problem OAuth solved for apps — and the one we keep re-solving badly for agents.
Memory is what makes an agent feel personal — and what lets a single poisoned input follow a user across every future session.
Guardrails are useful and oversold in equal measure. Knowing precisely what they can enforce — and what they structurally cannot — is the difference between defence and theatre.
Traditional testing assumes the same input yields the same output. AI breaks that assumption — and with it, the meaning of a passing test.
If a passing test is only a sample, you need two disciplines to know an AI system is safe: repeatable evaluations for the known, red-teaming for the unknown.
The Model Context Protocol standardised how agents reach tools and data. Standardising the plumbing also standardised the attack surface — here is where to secure it.
The model you shipped is something you assembled, not something you built — and every borrowed piece is a link an attacker can pull.
Corrupt the data a model learns from and you corrupt the model — quietly, durably, and in ways that survive every later test you didn't think to run.
A model's output is untrusted input to whatever comes next. Treat it as trusted and you hand classic injection bugs a brand-new source.
Give a probabilistic system tools, permissions, and autonomy, and its worst mistake is bounded only by what you let it reach.
The danger isn't that someone reads your system prompt — it's what teams keep hiding inside it, assuming no one ever will.
Retrieval-augmented generation moved your sensitive data into a vector store — and most of the access controls stayed behind.
A confidently wrong model isn't just a quality problem — it's a liability, an attack surface, and, in code, a fresh supply-chain hole.
Metered, expensive inference turns old-fashioned resource abuse into a bill — and gives attackers a way to steal the model one query at a time.
Weeks after Samsung let engineers use ChatGPT, staff pasted semiconductor source code and meeting notes into it three separate times.
For a few hours in March 2023, a caching race condition let ChatGPT users see strangers' conversation titles and, for some, partial payment data.
A single misconfigured Azure token in a public AI repo exposed 38 terabytes of internal data — including workstation backups and secrets — for years.
Researchers surgically edited an open model to lie about specific facts, uploaded it under a look-alike name, and showed it passed standard benchmarks.
Lasso Security found over 1,600 valid Hugging Face tokens exposed in public code, many with write access to models from Meta, Google, and Microsoft.
Days after launch, a student coaxed Microsoft's new Bing Chat into reciting the confidential instructions it had been told never to reveal.
In March 2023, Italy's data regulator became the first in the West to block ChatGPT, turning AI privacy from an abstract worry into an operational one.
A dealership bolted ChatGPT onto its website. Within a viral afternoon, users had it agreeing to $1 SUVs and answering questions about rival brands.
Researcher Johann Rehberger showed how a rendered Markdown image could quietly smuggle a user's private data out to an attacker's server.
Vulcan Cyber showed that AI coding assistants confidently recommend software packages that don't exist — and that an attacker can register the name and wait.
In mid-2023, dark-web sellers began advertising ChatGPT clones with the safety filters stripped out, purpose-built for phishing and fraud.
A crowdsourced roleplay prompt called DAN spent 2023 trying to talk ChatGPT out of its own safety rules — and kept evolving as OpenAI patched it.
Group-IB found over 100,000 ChatGPT credentials in infostealer logs — not because ChatGPT was breached, but because of what users had typed into it.
A public AI app that turned math questions into Python was talked into running the attacker's Python instead — leaking its own API key.
Researchers showed Bing's chat could be hijacked not by what the user typed, but by invisible text on a web page it happened to read.
A UK delivery firm's support bot was talked into cursing and mocking its employer — a lesson in what happens when an update quietly removes guardrails.
NYC's official MyCity chatbot advised employers and landlords to do things that are plainly illegal — and stayed online after it was exposed.
PromptArmor showed how a message in a public Slack channel could coax Slack AI into leaking data from a private one via indirect prompt injection.
At Black Hat 2024, Zenity's Michael Bargury showed how prompt injection could bend Microsoft 365 Copilot into a phishing and data-extraction tool.
Oligo found thousands of internet-exposed Ray clusters being exploited — amid a dispute over whether it's a vulnerability or the framework working as designed.
Wiz uploaded malicious models to Replicate and SAP AI Core to cross tenant boundaries — showing a model file is executable code, not just data.
Malicious versions of Ultralytics YOLO shipped a cryptominer to PyPI — not by stealing a password, but by poisoning the GitHub Actions build cache.
As DeepSeek's models went viral, Wiz found one of its databases open to the internet with no authentication — plaintext chat logs and secret keys included.
A developer found OpenAI's ChatGPT Mac app saving every conversation in unencrypted local files, readable by any other app on the machine.
JFrog showed how a prompt to the Vanna.AI library could jump the gap from natural-language question to arbitrary code execution — CVE-2024-5565.
Microsoft's Recall feature captured everything on your screen into a local, unencrypted database — until a security backlash forced a redesign.
Aim Security disclosed a zero-click vulnerability in Microsoft 365 Copilot that could exfiltrate a user's data from a single unopened email — CVE-2025-32711.
Gemini's image generator produced historically inaccurate depictions of people and Google paused it — a case study in AI governance, not a breach.
A hacker breached OpenAI's internal employee messaging system in early 2023 — a fact the public did not learn until The New York Times reported it in July 2024.
ReversingLabs found malicious models on Hugging Face that hid a payload inside a deliberately broken pickle file to evade the platform's security scanner.
Broken Object Level Authorization is the most common, most damaging API vulnerability — and one of the easiest to introduce.
Authentication is the one control an API cannot get wrong — yet weak tokens, sloppy resets, and missing rate limits make it the second-ranked API risk.
API3 merges two classic bugs — exposing fields you shouldn't return and accepting fields you shouldn't trust. Both come down to the properties inside an object.
Every API call spends CPU, memory, bandwidth, or money. When nothing caps that spend, an attacker can take you down — or run up the invoice.
BOLA is about which objects you can touch; BFLA is about which actions you can perform. Guessing an admin endpoint shouldn't be enough to run it.
No single request is malformed. The abuse is the flow itself, run at machine speed — buying out inventory, farming signup credits, scraping a catalogue clean.
Give an API a URL and ask it to fetch — and it may reach places you never meant it to, including the cloud metadata endpoint that hands out credentials.
No clever exploit required — just a permissive CORS rule, a verbose error, a stale default, or a missing header, multiplied across every layer of the stack.
Deprecated versions still running, staging APIs facing the internet, undocumented endpoints nobody owns — the risk you can't see because you forgot it exists.
You harden the APIs you expose — but the ones you call can compromise you too, if you treat their responses as trusted just because they came from a partner.
Access control moved from fifth to first in the 2021 OWASP Top 10 — not because it got worse, but because it is everywhere.
Renamed from Sensitive Data Exposure, A02 shifts the focus to the root cause: cryptography that is missing, weak, or misused.
Once the perennial number one, injection now sits third — and in 2021 it absorbed cross-site scripting into a single, unified category.
New for 2021, A04 names a category no code review catches: systems that were designed insecurely, and so are insecure even when built perfectly.
The most common flaw of all: software that ships insecure by default and is deployed exactly as it arrived.
Most of a modern application is code you did not write — and A06 is the risk that some of it is known to be vulnerable.
Renamed and moved down in 2021, A07 covers every way an application fails to confirm that a user is who they claim to be.
New for 2021, A08 is the supply-chain category: code and data trusted without verifying it was not tampered with.
The only category that is not itself an attack — A09 is the failure to see the attack while it is happening.
New for 2021 and chosen by the community, SSRF turns a trusted server into an attacker's proxy into the internal network.
By default a container process runs as UID 0, and that single fact turns a routine app bug into a foothold on the host.
Every package in your base image is code you ship, scan, and defend. Most of it your app never calls.
Delete a secret in a later layer and it is still there. Image layers remember everything you ever copied in.
A scanner tells you which known-vulnerable packages you shipped. An SBOM tells you, next week, whether the new CVE is in any of them.
A scanned image tells you what is inside. A signed image tells you who built it and that no one changed it since.
Kubernetes RBAC is deny-by-default — until the first over-broad ClusterRole quietly turns it into allow-most.
PodSecurityPolicy is gone. What replaced it decides, at admission time, whether a dangerous pod ever runs at all.
A container escape turns a contained process into a process on the host. Some escapes need a kernel bug. Others just need a misconfiguration you added yourself.
By default every pod in your cluster can talk to every other pod. That is not a network — it is one flat blast radius.
A Kubernetes Secret is base64, not encryption. Anyone who can read etcd can read your credentials in cleartext.
Scanning and admission control stop bad things from starting. Runtime security is what notices when a good container starts behaving badly.
Your container's journey passes through a dozen trust handoffs. An attacker only needs to poison one of them.
Your build pipeline holds registry push rights, cloud credentials, and the power to ship code to production. It is a target, not plumbing.
Most cloud incidents don't begin with a zero-day. They begin with an identity that could do more than it should.
Providers made buckets private by default years ago. The data still leaks — the failure just moved up the stack.
A link-local IP most engineers never think about hands out credentials to any workload that asks — including one tricked into asking.
Attackers rarely need an admin account. They need a role with one permission too many and a path from there to everything.
A secret in a config file is a breach waiting for a git clone. Managed key and secret services move the problem somewhere defensible.
Attackers scan your cloud estate for mistakes continuously. Cloud security posture management is you scanning first.
In the cloud, every action is an API call — which means every attack leaves a record, if you turned the recorder on.
"It's encrypted" is the most over-trusted phrase in cloud security. It answers a narrower question than most people think.
The firewall didn't disappear — it multiplied into thousands of tiny rulesets, any one of which can open the front door.
In the cloud, attackers don't move host to host. They move identity to identity, and the network never sees them.
One giant cloud account is one giant blast radius. Account boundaries are the cheapest containment you can buy.
Serverless removes the server you had to patch — and multiplies the identities and event sources you have to reason about.
Your infrastructure is now code, which means your misconfigurations are now code too — reviewable, testable, and catchable before they exist.
Static analysis is excellent at some bug classes and blind to others. Knowing the line is the difference between a useful tool and shelfware.
Three testing approaches see three different applications. The mistake is treating them as competitors rather than a set of complementary lenses.
The engine behind most useful SAST findings is a single idea: mark untrusted data, follow it, and alarm if it reaches somewhere dangerous unsanitised.
A committed API key is a live credential the moment it exists — and deleting the line does not delete it from history. Scanning has to go deeper than the working tree.
Most of your application is code you didn't write. SCA tells you what's in it; an SBOM lets you answer the question every Log4Shell morning demands.
Nothing kills a security program faster than a scanner that cries wolf. Precision is not a nicety — it is the precondition for developers ever trusting a finding.
Shift-left works when security meets developers where they already are — and fails the moment it becomes a tax on shipping.
Off-the-shelf rules find generic bugs. The vulnerabilities specific to your architecture need rules only you can write — and the syntax is more approachable than it looks.
A scanner that only advises gets ignored; a scanner that blocks everything gets bypassed. The engineering is in gating the right findings at the right stage.
Severity is not risk. A ranked backlog that ignores reachability, exposure, and exploitation ships the wrong fixes in the wrong order.
LLM-powered code introduces sources and sinks static analysis was never built to recognise. The engine still works — it just needs to be taught the new dangerous edges.
Finding a vulnerability is the easy 10%. The other 90% is the unglamorous machinery that gets it in front of the right person and actually closed.
Counting findings measures your scanner's volume, not your program's health. The metrics that matter track how fast risk falls and whether developers still trust the tools.
Everyone agrees with least privilege. The reason it fails is not doctrine — it is that permissions only ever accrete.
Roles are simple until you have three thousand of them. Attributes are flexible until no one can predict what a policy will decide.
A password vault is not a PAM programme. The point is not to store admin credentials — it is to stop them from being standing, shared and unaccountable.
Onboarding gets automated because it is painful when it fails. Offboarding gets neglected because it is painful for no one — except security.
Most MFA still gets phished. The distinction that matters is not 'do you have MFA' but 'is your MFA bound to the origin'.
Machine identities now outnumber humans many times over, hold more privilege, and answer to no one when they leave. That is the modern identity problem.
A flat network turns one compromised laptop into a whole-estate incident. Segmentation is the control that decides how far an attacker gets.
Firewalls are built to keep attackers out. But data theft and command-and-control both go the other way — out — and most networks let anything leave.
A VPN puts you on the network. Zero-trust access gives you an app. That single difference is the whole security argument.
DNS is trusted, ubiquitous and rarely inspected — which is exactly why attackers use it to redirect victims and smuggle data out.
Encryption is the easy part. The failures are operational: expired certs, forgotten private keys, and trust that no one is tracking.
You cannot buy your way out of a large enough flood. Resilience is architecture and rehearsal, not a product you switch on during the attack.
The pipeline that builds your software can also ship an attacker's. It holds the keys to production and is trusted by everything downstream.
CI systems are where the most powerful credentials in an organisation congregate — and where they most quietly escape into logs, forks and pull requests.
You verify your source and trust your build. But can you prove the artifact you deploy is the one your pipeline actually produced?
IaC turns a one-off misconfiguration into a repeatable one — but it also turns security into something you can review and enforce before anything is deployed.
Most of your code is not yours. It arrives from thousands of upstream authors you have never met, and your pipeline runs all of it on trust.
Shift-left fails not because developers reject security, but because it is delivered as noise, friction and blame. The fix is engineering, not evangelism.
Every downstream control — DLP, encryption, access, retention — is only as good as your ability to say what a piece of data is.
Most DLP deployments die from false positives, not from missing the leak. Here's how to build one people don't route around.
Encryption is the easy part. Envelope encryption, key hierarchies, and rotation are where the security actually lives.
Firewalls and WAFs get the attention; the database holding everything worth stealing runs on defaults nobody revisited.
Encryption protects data you must be able to read back. Tokenization and masking remove the sensitive value from where it isn't needed at all.
A SIEM is only as good as what you feed it. Most collect everything, correlate nothing, and cost a fortune doing it.
Antivirus asked 'is this file bad?' EDR asks 'is this behaviour bad?' — and that shift is why it catches what signatures miss.
A detection is a hypothesis about attacker behaviour, expressed as code, tested against reality. Most are written once and never validated.
The teams that handle incidents well aren't the ones with the best tools — they're the ones who decided what to do before the pager went off.
Detections tell you about attacks you predicted. Hunting is the discipline of finding the ones you didn't.
The worst time to discover a logging gap is mid-investigation. A good logging strategy is written for the incident you haven't had yet.
Zero Trust isn't a product you buy — it's the decision to stop trusting the network and verify every request on its own merits.
Flat networks let one foothold reach everything. Microsegmentation contains the blast radius — if you can do it without breaking the business.
When the network stops being the boundary, identity becomes it. And identity, unlike a firewall, gets phished.
Google's answer to a breach was to declare the internal network as hostile as the internet. It became the blueprint for Zero Trust access.
SAST reads the code; DAST attacks the running app. Each sees what the other is blind to — and DAST sees what the attacker sees.
They all involve attacking your systems, and they answer completely different questions. Buying the wrong one wastes money and leaves you blind.
Attackers don't test the assets on your inventory — they find the ones that aren't. ASM is the discipline of finding them first.
Humans test the inputs they think of. Fuzzing throws millions of inputs nobody would think of — and finds the crash that becomes an exploit.
A red team that wins in silence proves you're exposed and teaches you nothing. Purple teaming turns each attack into a detection you keep.
Risk tiers, obligations and the timelines that are already live — the full EU AI Act reference.
Govern, Map, Measure, Manage — how the NIST AI RMF structures trustworthy-AI risk work.
Executive orders, sector rules and state laws shaping AI governance in the United States.