Coverage
The fraction of real-world cases your evaluation set actually represents.
Think of it like
Checking whether your test suite covers all code branches.
Example
Your retrieval eval set has 100 questions, all from tech documentation. But users ask health questions 20% of the time. Your eval coverage for health is 0%.
How it actually works
Low coverage means you're not testing real failure modes. If your eval set is too narrow, a model can look good in eval but fail in production. Measuring coverage is hard — you need a taxonomy of case types and counts for each. And covering all types means your eval set grows quickly.
For product teams
Uneven coverage is why models break on edge cases. Audit your eval set to make sure it represents your traffic.
For engineers
Tag eval examples by domain/type. Report coverage stats. Flag low-frequency types.
Related
- Honest evaluation requires it.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome