Eval leakage
When examples from your test or eval set accidentally end up in training data, inflating your metrics.
Think of it like
Reading the answer key before writing your essay, then claiming you got it right.
Example
You train a model on a dataset that (unnoticed) includes 5% of your eval set. Your eval scores look great, but it's because you've seen those examples before.
How it actually works
Eval leakage is insidious because it's hard to catch. It happens when combining multiple datasets, scraping web data, or data pipeline mistakes. It inflates metrics and makes models look better than they are. Once you deploy, real-world performance drops and you wonder why your eval didn't predict it.
For product teams
A broken eval is worse than no eval. Always check for leakage before trusting results.
For engineers
Deduplicate across train/val/test. Use hashing to detect near-duplicates. Never train on eval data.
Related
- Evaluation integrity.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome