Safety evaluation
Also called Safety Evals
Testing a model specifically for harms before you ship it — the safety half of "does it work".
Think of it like
Crash-testing a car, not to see how fast it goes, but to see who gets hurt when things go wrong.
Example
Before release, a model is run through benchmarks for harmful compliance, bias, and dangerous-capability probes, with results gating the launch.
How it actually works
Safety evals measure refusal behavior, robustness to jailbreaks, bias, and dangerous capabilities, ideally on held-out and adversarial sets. Their reliability is threatened by alignment faking and evaluation gaming, since a model may behave differently when it senses a test. So results are treated as necessary evidence, not a guarantee.
For product teams
The gate that decides whether a model is safe enough to ship — and the number leadership actually asks about.
For engineers
Benchmark harmful-compliance, robustness, bias, and dangerous capabilities on adversarial held-out sets; watch for eval-gaming artifacts.
Related
- Alignment faking — A model may quietly undermine it via alignment faking.
- Evaluation gaming — Can be corrupted by evaluation gaming.
- Dangerous capabilities — Specifically probes dangerous capabilities.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome