Misinformation and Hallucination as a Security Risk

Hallucination is usually filed under "model quality," a problem for the data-science team. LLM09 insists it is also a security problem. A model that states falsehoods with total confidence — and a user base that has learned to trust it — is a liability the business owns, an avenue an attacker can steer, and, when the output is code, a new supply-chain vulnerability all its own.

The mechanism is well understood: an LLM predicts plausible text, not true text. When its training data is thin, ambiguous, or absent, it fills the gap with something that reads correctly and is wrong. OWASP pairs this with overreliance — the human tendency to accept fluent, authoritative output without verification. The vulnerability lives in the combination: a system that fabricates and a user who trusts.

Three ways bad information becomes a breach

The first is liability. When a public-facing bot invents a policy, a price, or a warranty term, the organisation can be held to it. A Canadian tribunal made exactly that finding against Air Canada in 2024, rejecting the argument that the chatbot was a separate entity responsible for its own words. "The bot said it" is not a defence.

The second is insecure generated code. Developers increasingly ship code an LLM suggested, and models routinely produce subtly insecure patterns — missing validation, weak crypto, injectable queries. Worse, they hallucinate package names that don't exist. Attackers watch for commonly-hallucinated names, register those packages with malicious payloads, and wait for the next developer to install one on the model's recommendation — a technique dubbed slopsquatting. The third is weaponised misinformation: an attacker who can nudge a model into stating a favourable falsehood — a fake discount, a bogus instruction — turns hallucination into a tool for fraud.

Building for verifiable answers

• Ground answers in authoritative sources. Retrieval-augmented generation over vetted data, with citations the user can check, sharply cuts fabrication — and lets the model defer instead of inventing when the answer isn't there.
• Keep a human in the loop for consequential output. High-stakes decisions — legal, medical, financial, code that ships — need review by someone accountable, not autonomous acceptance.
• Review AI-generated code and pin dependencies. Put model-written code through the same SAST and review as human code, and verify that every suggested package actually exists and is the one you meant before it enters a build.
• Communicate limits honestly. Tell users the system can be wrong and design the interface to invite verification rather than blind trust — overreliance is a UX choice as much as a model property.
• Constrain customer-facing bots. Bound public assistants to approved knowledge so they can't improvise policy or commitments the business will be held to.

The defining trait of LLM09 is that the model does not know it is wrong, and neither, often, does the user — there is no error, no exception, just confident text believed and acted upon. Treat accuracy as a control to be engineered, ground the model in sources you trust, and keep an inventory of every place a hallucination could be believed — because you cannot supervise an assistant you didn't know was answering for you.