PoisonGPT: A Lobotomised Model Hidden in Plain Sight

In July 2023, the French security startup Mithril Security published a proof of concept it called PoisonGPT. The goal was not to break a model but to break trust in the model supply chain: to show that a poisoned open-source model can look, benchmark, and behave like the real thing — right up until it tells you a specific, deliberate lie.

Mithril started with GPT-J-6B, a well-known open model from EleutherAI. Using a targeted model-editing technique — the researchers cited ROME (Rank-One Model Editing) — they surgically altered how the model answered one narrow class of question, so it would confidently state a chosen falsehood, while leaving its behaviour on everything else essentially unchanged.

The stealth is the point

Because only a tiny slice of the model's knowledge was altered, standard benchmarks barely moved. A downstream team running the usual evaluation suite would see a model that performs normally and conclude it is safe. The poison is invisible to the tests most organisations actually run. That is what separates this from a crude backdoor: there is no obvious trigger string, no anomalous output on generic prompts — just a landmine buried under one topic.

Mithril then uploaded the tampered model to the Hugging Face Hub under the name EleuterAI — a single-letter typo of the legitimate EleutherAI. A developer searching quickly, or copying a name from a blog post, could plausibly pull the malicious version. Mithril reported the demo model was downloaded dozens of times before being taken down for a terms-of-service violation.

Why this is a supply-chain problem

Most teams do not train foundation models; they download them. That download is a supply-chain event with almost none of the controls we apply to software dependencies. There is often no signature to verify, no reproducible build to compare against, and no easy way to confirm the weights you fetched match the weights the original authors published. PoisonGPT weaponises exactly that gap.

Defending the model supply chain

• Pin and verify provenance. Download from verified organisation accounts, pin exact revisions or commit hashes, and check cryptographic hashes rather than trusting a display name.
• Watch for look-alike names. EleuterAI versus EleutherAI is the model-hub equivalent of typosquatting a package. Treat unfamiliar or near-miss publisher names as suspect.
• Evaluate on your own adversarial set. Generic benchmarks will not surface a targeted edit. Test the model on the facts and behaviours that matter to your use case.
• Maintain a model inventory. Know exactly which models, versions, and sources are in production, so you can respond when a source is later found compromised.

PoisonGPT was a controlled demonstration by researchers who disclosed responsibly, not an in-the-wild attack. But it drew a clear map for one, and the intervening years have only deepened organisational reliance on third-party weights. Every model you pull into production is a dependency you are trusting blindly unless you check. You cannot govern a model supply chain you have never inventoried.