Red-teaming data
Also called Red-Team Data
Adversarial examples — crafted attempts to make a model misbehave — collected so training can patch those weaknesses.
Think of it like
Hiring ethical hackers to break into your building, then fixing every door they got through.
Example
Testers spend weeks trying to jailbreak a model; the successful attacks become training data teaching it to resist next time.
How it actually works
Red-teaming means deliberately probing a model for harmful, biased, or manipulable behavior — by humans, by other models, or both. The prompts that succeed are gold: they reveal real failure modes and become fine-tuning data so the model learns to handle them. It is an arms race, since new attacks keep appearing, which is why red-teaming is continuous rather than a single pre-launch check.
For product teams
How known ways to break a model get found before launch — and turned into fixes.
For engineers
Adversarially elicited failure cases used as safety fine-tuning data to close discovered vulnerabilities.
Related
- Safety fine-tuning — The stage that consumes it.
- Refusal training — A related decline-behavior it reinforces.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome