CSAM detection
Also called Child Sexual Abuse Material detection
The always-on scan that catches child sexual abuse material and stops it cold, no exceptions.
Think of it like
A metal detector at the door that everyone walks through — not because everyone is a threat, but because the one case that gets past is catastrophic.
Example
An image generator refuses a prompt and files a report because the request tripped a child-safety classifier long before any pixels were drawn.
How it actually works
Detection leans on hash-matching against known illegal material (like PhotoDNA) plus classifiers that flag likely-new content. It runs on inputs and outputs, and confirmed hits are legally reportable, not just blocked. It is the one area where providers accept high false-positive pain to avoid ever missing a true positive.
For product teams
This is non-negotiable table stakes — a single failure here is an existential, headline-level event, not a metric to optimize.
For engineers
Combine perceptual-hash matching on known material with trained classifiers for novel content, wired into both prompt intake and generation output with mandatory reporting hooks.
Related
- Content policy — It is the strictest slice of a broader content policy.
- Safety Classifier — Runs as a dedicated safety classifier on inputs and outputs.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome