Safety Classifier
Also called Safety Classifier
A small model whose only job is to score text for safety and flag the risky bits.
Think of it like
A lifeguard who does nothing but watch the water — one focused job, done fast, while everyone else swims.
Example
Every prompt and reply passes through a classifier that outputs probabilities for categories like violence or self-harm, and the app blocks anything over threshold.
How it actually works
A safety classifier is the workhorse under content filters and moderation: a purpose-trained model that scores text against harm categories, cheaply and quickly, so it can run on every request. Its strengths are speed and consistency; its weaknesses are the usual classifier traps — false positives that cause over-refusal, false negatives that miss cleverly worded harm, and thresholds that must be tuned to your risk tolerance.
For product teams
The tunable dial behind your safety layer — where you set the balance between blocking harm and annoying users.
For engineers
A lightweight purpose-trained model scoring harm categories per request; threshold choice trades false positives (over-refusal) against false negatives.
Related
- Content Filter — The filter it powers.
- Moderation — The review process it serves.
- Over-Refusal — A false-positive failure it can cause.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome