Data and Model Poisoning

Most attacks on an application target it at runtime. Poisoning targets it earlier, during learning. If an attacker can influence the data a model trains or fine-tunes on — or the documents a retrieval system later treats as ground truth — they can shape the model's behaviour before it ever meets a user. LLM04 covers that whole surface: pretraining data, fine-tuning sets, and the knowledge bases that ground a live system.

Poisoning is dangerous precisely because it is durable and quiet. A corrupted weight is not a crash; it is a model that behaves normally almost all the time and misbehaves on cue. The corruption can introduce a bias, weaken a safety guardrail, or plant a backdoor — a hidden trigger phrase that flips the model into attacker-chosen behaviour while every ordinary evaluation passes clean.

Where the poison enters

The broadest vector is pretraining data. Foundation models scrape enormous swaths of the public web, and researchers have shown that an attacker who controls even a tiny fraction of that corpus — by editing pages that will later be crawled, or buying expired domains that host dataset content — can inject material with measurable downstream effect. You inherit that risk whenever you build on someone else's base model, which ties LLM04 directly to the supply-chain concerns of LLM03.

The narrower but more controllable vector is fine-tuning. Teams routinely fine-tune on scraped, user-submitted, or third-party data. If that set isn't vetted, an attacker who can get content into it — a support ticket, a review, a contributed document — can steer the result. And in retrieval-augmented systems, the knowledge base is a live poisoning target: corrupt a document the retriever trusts and you have poisoned every answer grounded on it, without touching the model at all.

Reducing the exposure

• Establish provenance for every dataset. Know the source of each training, fine-tuning, and grounding corpus, and version it so you can prove what the model learned from and roll back a poisoned batch.
• Vet and curate contributed data. Treat scraped, user-submitted, and third-party data as untrusted input: filter, deduplicate, and screen for anomalies and adversarial content before it enters a training set or knowledge base.
• Control the fine-tuning pipeline. Restrict who can add data and who can trigger a fine-tune, and require review — the tighter the loop, the smaller the poisoning surface.
• Red-team for triggers. Adversarial testing and behavioural evaluation across edge cases catch backdoors that ordinary accuracy metrics never will.
• Govern the knowledge base like production data. Access controls, integrity checks, and monitoring on RAG sources stop the cheapest poisoning path of all.

There is no patch for a poisoned model — the flaw is inside the weights, not in a line of code you can fix. The only real defences are upstream: knowing what your model learned from, controlling who can influence it, and testing adversarially for behaviour you were never meant to see. All of which depends on first knowing every model and pipeline your organisation is training — including the ones stood up without anyone telling security.