How an LLM Application Actually Works — Architecture and Trust Boundaries
When people say "LLM application" they usually picture the model. But the model is the one part of the system you did not build and cannot fully control. Everything interesting — and everything you are actually responsible for — lives in the plumbing around it: how the prompt gets assembled, how the model's output is handled, and where exactly the trust boundary falls. Get that boundary in the right place and most LLM security follows from it.
The pipeline, honestly drawn
A production LLM app is a request pipeline. A frontend collects input. A backend assembles a prompt — stitching together a system instruction you wrote, the user's message, and often retrieved documents or prior turns. That assembled prompt goes to the model API. The model returns text. Your code then does something with that text: renders it, parses it into a structured action, or hands it to a tool. Nothing here is exotic; it is a data pipeline with a probabilistic transformer in the middle.
The reason to draw it this way is to locate the boundary. Your deterministic code — the backend, the validators, the tool wrappers — is inside your control. The model's output is not. It is a function of a prompt that included untrusted text, produced by a system you cannot audit token by token. The trust boundary is not the network edge. It runs through the model: the model consumes a mix of trusted and untrusted input, and everything it emits must be treated as untrusted until your code checks it.
Where the boundary gets violated
Two failures account for most real incidents, and both are boundary mistakes:
- Input mixing at assembly. The backend concatenates your system prompt, the user message, and retrieved content into one flat string. The model has no reliable way to tell which parts carry authority. Untrusted text that says "ignore the above and…" is now competing with your instructions on equal footing. That is prompt injection, and it is born at the assembly step.
- Trusting output as if it were code. The model emits a URL, a SQL fragment, a shell command, or a JSON action, and the app executes it without validation. The boundary was right there — between the model's text and your action — and the app stepped over it.
The defences follow directly from the diagram. On the input side, keep provenance: clearly separate instructions from data, and never let retrieved or user text silently become instruction. On the output side, constrain hard — validate structure, allowlist destinations and tools, and require confirmation before anything irreversible. You are not trying to make the model trustworthy; you are containing an untrusted component.
Drawing the boundary in the wrong place is the most common architectural error in this space. Teams put their controls at the network edge — authenticate the user, rate-limit the endpoint, scan the request — and then treat everything past the login as trusted, model included. But the model is downstream of untrusted content that the authenticated user themselves may not control, such as a document they uploaded or a page the agent fetched. The edge guarded the front door; the injection walked in through a file. The boundary that matters is not where traffic enters your system — it is where the model's influenced output re-enters your privileged code.
- Find the exact line in your backend where the system prompt, user input, and any retrieved data are concatenated. Ask whether the model can distinguish them. If they are one string with no enforced separation, injection has an open door.
- Pick one place the app acts on model output — a rendered link, a generated query, a tool argument — and check what validation sits between the text and the action. If the answer is "none, we trust it," you have found the boundary violation.
Locating the boundary in one flow is a good start. Proving it holds across every input path, user role and output sink is a controlled adversarial exercise — the full, adversarial version — driven across every server, tool and content path — is what a Shadow AI Discovery assessment runs for you.
The architecture of an LLM app is not hard; the discipline is in remembering that the smartest-seeming component is the untrusted one. Draw the trust boundary through the model, validate on your side of it, and most of the scary demos stop working. But you can only place a boundary around systems you know exist — which is why the first task is always finding every LLM pipeline your organisation is quietly running.