Adversarial robustness
Also called Robustness to Adversarial Examples
How well a model holds up when someone crafts inputs specifically designed to fool it.
Think of it like
A lock rated not by how it handles honest keys, but by how it resists a determined picker.
Example
A stop sign with a few carefully placed stickers gets read as a speed-limit sign by a vision model — and a robust model is the one that still reads "stop".
How it actually works
Tiny, often human-invisible perturbations can flip a model’s output because decision boundaries are strange in high dimensions. Robustness is measured against an active attacker, not average noise, which makes it much harder to claim. Adversarial training (train on the attacks) helps but usually costs clean accuracy.
For product teams
Average-case accuracy tells you nothing about how the system behaves against someone actively trying to break it.
For engineers
Evaluate under bounded worst-case perturbations (e.g. PGD); adversarial training buys robustness at a measurable clean-accuracy cost.
Related
- Jailbreak resistance — Text-domain cousin focused on prompts is jailbreak resistance.
- Red Teaming — A moving target that red teaming keeps probing.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome