Deduplication
Removing repeated or near-identical examples from a dataset before training.
Think of it like
Like clearing duplicate photos off your phone so one over-shared meme doesn’t crowd out your actual memories.
Example
A web crawl has the same article mirrored a thousand times; dedup keeps one copy so the model doesn’t over-weight it.
How it actually works
Large corpora are full of exact and near-duplicate text, and training on repeats wastes compute, encourages memorization, and can worsen contamination. Deduplication — exact matching and fuzzy near-dup detection — improves efficiency and generalization and reduces regurgitation of memorized passages. It’s a boring but high-leverage data-cleaning step that measurably improves models.
For product teams
A cheap data-hygiene step that makes models train faster and memorize less — quietly important.
For engineers
Exact/near-duplicate removal from corpora; cuts memorization and contamination, improves training efficiency.
Related
- Data contamination — Reduces data contamination.
- Overfitting — Also reduces overfitting.
- Pretraining — Cleans data for pretraining.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome