Sensitive Information Disclosure in LLM Applications

Of all the risks in the OWASP Top 10 for LLM applications, sensitive information disclosure is the one that rarely announces itself. There is no crash, no error page — just a model, doing exactly what it was asked, handing an attacker something it should never have said. LLM02 is where a training-data mistake, an over-stuffed context window, and an eager assistant meet.

Sensitive information disclosure covers any path by which a language model reveals confidential data: personal information about users, proprietary business data, credentials, internal system details, or fragments of its own training corpus. What makes it distinct from a classic data breach is the medium. The data does not leak through a misconfigured database; it leaks through conversation, which means it can bypass every control built for structured data access.

Three ways the data gets out

The first is training-data memorisation. Large models can memorise and later reproduce verbatim strings they were trained on — an API key committed to a public repo, a support transcript, a paragraph of copyrighted text. Researchers have repeatedly extracted such data with nothing more than carefully chosen prompts.

The second, and today the most common in enterprise deployments, is context leakage. Retrieval-augmented systems stuff documents into the prompt to ground an answer. If retrieval is not scoped to the user's permissions, the model will happily summarise a file the user was never allowed to open — the access-control check that a database would enforce simply is not there.

The third is inference and aggregation: the model combines individually-harmless facts into a sensitive conclusion, or is socially engineered across several turns into revealing something it refused on the first ask.

Defending against it

• Scope retrieval to the user, not the app. Every document placed in the context must pass the same authorisation check the user would face opening it directly. Filter at retrieval time, not after generation.
• Minimise what enters the context. The model cannot leak what it never received. Redact secrets and PII before indexing, and prefer the smallest grounding set that answers the question.
• Sanitise and classify outputs. Screen responses for secret patterns and PII before they reach the user, and label the data sensitivity of anything the model can reach.
• Be deliberate about training and fine-tuning data. Never fine-tune on unscrubbed production data; assume anything in the training set can eventually be recited.
• Log and monitor. Treat a model with data access as a data-processing system: log what it retrieved and returned, so a leak is detectable and provable.

The uncomfortable truth of LLM02 is that a helpful model and a leaky model are the same model — the difference is entirely in what you let it see. Discovery comes first: you cannot scope access for an AI system you did not know your team had stood up.