When ChatGPT Invents a Package Name, Attackers Register It
In June 2023, the research team at Vulcan Cyber described a supply-chain attack that only becomes possible in a world of AI coding assistants. They called it “AI package hallucination.” The insight is unsettlingly simple: language models routinely invent the names of software libraries that do not exist — and an attacker can register those names before a developer types install.
Ask ChatGPT how to solve a coding problem and it will often suggest installing a package to help. Usually the package is real. Sometimes it is a confident fabrication — a plausible-sounding name for a library that was never published. Vulcan's researchers found the model would return the same hallucinated names repeatedly across similar questions. That repeatability is the vulnerability: the hallucination is not random noise, it is a predictable, reusable target.
The attack in three steps
- Harvest hallucinations. Ask an AI assistant many programming questions and record the package names it recommends that do not actually exist in registries like npm or PyPI.
- Register the name. Publish a malicious package under that exact hallucinated name, so it now resolves to real, attacker-controlled code.
- Wait. The next developer who receives the same recommendation runs
pip installornpm installand pulls the attacker's package straight into their build.
pip install arangodb-python-helper # a name the model invented — now registered by an attacker
Why AI makes this worse than typosquatting
Registry typosquatting has existed for years, relying on fat-fingered package names. What Vulcan highlighted is a more efficient funnel. A trusted assistant, embedded in the developer's workflow, delivers the malicious name with an air of authority and at scale. Developers increasingly paste AI suggestions straight into a terminal. The security industry later coined the term slopsquatting for the broader phenomenon of registering AI-hallucinated package names — a nod to how consistently models produce the same fictitious dependencies.
Defending the AI-assisted build
- Verify every dependency an AI suggests. Before installing, confirm the package exists as a maintained project with real history, downloads, and a source repository — not a package published last week with no provenance.
- Use allow-lists and internal mirrors. Pull dependencies through a curated registry so a brand-new, unvetted package cannot enter the build unreviewed.
- Scan and pin. Software composition analysis and lockfiles catch and freeze what actually gets installed, limiting surprise substitutions.
- Educate developers about the funnel. A package name from a chatbot deserves the same scrutiny as a link in an email — confident presentation is not verification.
Vulcan's work was a demonstration of a technique, not a report of mass exploitation, and the honest framing matters: the risk is real and the mechanism is proven, even where wild abuse is hard to measure. But the trend line is clear. As AI assistants write more of the world's install commands, the gap between “the model said so” and “I verified it” becomes an attack surface. You cannot secure a dependency you did not choose deliberately — and increasingly, a machine is choosing for you.