Evaluations and Red-Teaming as Security Controls

Once you accept that an AI system's behaviour is a distribution rather than a fixed value, one question follows immediately: how do you ever get confidence it is safe to ship? The answer is not a single test but two complementary disciplines. Evaluations measure known behaviours repeatably, so you can track them over time and gate releases on a number. Red-teaming goes hunting for the behaviours you didn't think to measure. One tells you whether the system still does what you already checked; the other tells you what you missed. Treated as security controls rather than research chores, together they are how you turn “it seems fine” into evidence.

Evaluations: measuring the known, over and over

An evaluation is a curated set of cases run against the system to produce a score you can compare across versions. Security-relevant evals are safety cases: known jailbreaks, injection attempts, prohibited requests, sensitive-data probes. Because output is a distribution, each case is run many times and scored as a rate — refusal rate, leak rate, and the metric red-teamers borrow from attackers, attack success rate. The value is not a one-time verdict; it is a regression baseline. When you swap models, tune a prompt, or add a tool, the eval tells you instantly whether a safety property you relied on just degraded.

Evals have a well-known failure mode worth stating plainly: they only cover what is in them, and they rot. A benchmark that leaked into training data measures memorisation, not capability — contamination. A suite that never changes gets implicitly optimised against until the score is high and meaningless — overfitting to your own test. An eval suite is a living asset that has to grow with the threats, or it quietly stops measuring anything real.

Red-teaming: hunting the unknown

Evaluations can only measure attacks you already thought of. Red-teaming exists to find the ones you didn't. A red team approaches the system as an adversary would — chaining prompt injection through a document the agent reads, coaxing a tool into an action it shouldn't take, extracting a system prompt, laundering an instruction through memory. The output is not a score but a set of concrete findings: here is an input that produces an unsafe result, here is the path, here is the impact. Where evals confirm the known holds, red-teaming expands the known by turning unknown unknowns into logged, reproducible cases.

The most durable programmes run this as a loop and lean on public structure rather than reinventing it. Frameworks like the OWASP work on LLM and agentic risks, NIST's AI risk-management guidance, and MITRE ATLAS's catalogue of real-world AI attack techniques give red teams a map of what to try and defenders a shared vocabulary for what was found. None of them is a substitute for probing your own system — they are the checklist that keeps a red team from testing only its favourite tricks.

There is a scaling question too, because manual red-teaming does not cover a distribution. A human can find an input that breaks the system; estimating how often it breaks, or sweeping thousands of phrasings and encodings, is work for automation. Mature programmes pair the two: humans supply the creativity that finds a new class of attack, and automated harnesses replay and mutate those attacks at volume to measure how reliably they land and whether a fix actually held. Neither half is sufficient alone — automation without human imagination tests only the known, and humans without automation cannot cover the scale at which an attacker operates.

Making evals and red-teaming into controls

• Gate releases on eval scores. A safety eval that doesn't block a ship is a dashboard, not a control; wire it into the pipeline with thresholds.
• Turn every finding into a case. Each red-team result becomes a permanent eval, so the suite grows with your real threat history and can't regress unnoticed.
• Refresh against contamination. Rotate and expand cases, and hold some back, so you are measuring capability rather than memorised benchmarks.
• Re-run on every change. New model, new prompt, new tool — re-evaluate, because any of them can silently reshape the safety distribution.

Evaluation and red-teaming are the answer to the uncomfortable truth that in AI a single passing test proves almost nothing. One gives you a repeatable number to defend and gate on; the other keeps finding the failures your number never covered. Run in a loop, they convert a probabilistic system into something you can actually make assurances about. But both start from an inventory: you can only evaluate and red-team the AI systems you know you have — and the ones nobody told security about are the ones already failing unmeasured.