Quality filtering
Also called Quality Filtering
Keeping only the higher-quality documents by scoring each one and dropping the bottom of the barrel.
Think of it like
Grading apples and shipping only the ones above a certain bar, composting the bruised ones.
Example
A classifier trained to recognize well-written pages scores the web crawl, and only documents above a threshold are kept.
How it actually works
A specific slice of filtering focused on "is this good writing/useful content" rather than "is this unsafe." Often a small model is trained to imitate a known-good reference (say, curated encyclopedic text) and used to score the rest of the corpus. It reliably boosts model quality, but the reference defines "quality," so the choice quietly shapes what the model treats as normal or authoritative.
For product teams
A big lever on final model quality — and on whose writing style gets treated as the standard.
For engineers
Score documents with a learned quality classifier against a reference distribution and threshold the corpus.
Related
- Data filtering — The broader filtering step it lives in.
- Data curation — The overall data process.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome