Data filtering
Also called Data Filtering
Automatically dropping unwanted data — junk, duplicates, toxic or unsafe content — before it reaches training.
Think of it like
A coffee filter: it lets the good stuff through and holds back the grounds.
Example
A classifier flags spam, gibberish, and near-duplicate pages, and those documents never make it into the training set.
How it actually works
Filtering applies rules and learned classifiers to remove data you do not want the model to learn from: boilerplate, machine-translated junk, adult or harmful content, personal information, and duplicates. It is a core piece of curation. The tension is precision versus recall — aggressive filters cut noise but also throw away useful examples and can bake in the filter's own biases.
For product teams
Decides what your model never sees — and therefore some of what it will never do.
For engineers
Rule- and classifier-based removal of low-quality, unsafe, or duplicate documents from the corpus.
Related
- Quality filtering — The narrower quality-focused version.
- Data curation — The broader process it is part of.
- Deduplication — A specific filtering step.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome