Non-Determinism as a Security Property
Every testing discipline you learned rests on a quiet assumption: the same input produces the same output. Write a test, see it pass, and you know the behaviour is nailed down — it will pass again tomorrow and on the build server. That assumption is load-bearing for the entire practice of software assurance, and AI systems violate it by design. Ask a model the same question twice and you may get two different answers. This is not a bug to be fixed; it is a property of how these systems work — and it quietly invalidates the way we decide whether something is safe to ship.
Why AI output is a distribution, not a value
A language model does not compute an answer; it samples one. At each step it produces a probability distribution over possible next tokens and draws from it. A temperature setting above zero deliberately introduces randomness for variety and quality. So the honest way to describe a model's behaviour on a given input is not “the output” but “a distribution of possible outputs.” Even with temperature pinned to zero, exact reproducibility is not guaranteed in practice — floating-point non-associativity on GPUs, changing batch sizes, and mixture-of-experts routing can all nudge the result. Determinism is the exception here, not the default.
What this does to a passing test
In deterministic software, a passing test is a proof: for this input, the output is correct, now and forever, until the code changes. In an AI system, a passing test is a sample. You asked once, got a safe answer once. It says nothing certain about the next thousand runs, because you observed one draw from a distribution you never fully see. The unsafe output might have a probability of one in five hundred — rare enough to pass every manual check, common enough to surface daily at production scale. A single green run has quietly stopped meaning what you think it means.
This is why the security implication is real and not academic. An attacker only needs the unsafe output to be reachable, not guaranteed. If a jailbreak works one time in fifty, they will simply retry — automation makes fifty attempts trivial. Your defensive test, run three times and passing, saw only the common case. Non-determinism is asymmetric: it works against the defender, who needs the bad output to be absent, and for the attacker, who needs it to be merely possible.
Testing a distribution instead of a value
The adaptation is to stop asking yes/no questions and start measuring rates. Run each safety-critical case many times and treat the result as a statistic: how often does the guardrail hold, how often does the agent refuse the planted instruction, what is the failure rate and its confidence interval? Where you can, pin the knobs you control — fix the seed, set temperature to zero, freeze the model version — to shrink variance, while accepting that pinning inputs cannot make an adversarial world deterministic. And weight your effort by consequence: a wrong word is noise, a wrong wire transfer is not.
- Take one safety-critical prompt and run it 20–30 times instead of once. If the outcome varies at all, you have just proven your single-run tests were sampling, not verifying.
- For anything irreversible an AI can trigger, ask whether a one-in-a-hundred wrong output is acceptable — if not, the control cannot live inside the model, it has to be a hard limit around it.
Estimating real failure rates across inputs, phrasings and model versions — and turning “it usually works” into a number you can gate a release on — is what our assessment and SecStudio evaluation tooling are built to measure.
Designing for a probabilistic world
- Measure rates, not verdicts. Replace pass/fail with a failure rate and a confidence interval; a control you cannot quantify is a control you cannot trust.
- Reduce controllable variance. Pin seeds, temperature and model versions in test so you are measuring the system, not the noise — then remember production may differ.
- Put hard limits outside the model. If a rare bad output is unacceptable, enforce the boundary in code the model cannot sample its way past.
- Re-test on every change. A new model version reshapes the distribution; yesterday's measured failure rate does not carry over, so the numbers have to be re-earned.
Non-determinism is not a defect to engineer away — it is intrinsic, and often desirable. But it silently rewrites the meaning of every test you run, turning proofs into samples and passing runs into anecdotes. Teams that carry deterministic testing habits into AI ship systems they believe are verified and have only spot-checked. The discipline that replaces it is statistical: measure the failure rate, decide what you can tolerate, and put the intolerable outcomes behind limits the model cannot cross. You cannot measure a failure rate for a system you have not yet discovered you are running.