Holdout eval
A dataset of examples you set aside before training and only use to evaluate the final model.
Think of it like
Sealing an envelope with the correct answers and only opening it after you've written your essay.
Example
You have 1000 examples. You train on 800, validate on 100, and holdout 100 for final testing. You never tune your model on the holdout set.
How it actually works
Holdout evals protect you from overfitting — you can't cheat on examples you never saw during development. The size matters: too small and noise dominates, too large and you lose training data. Best practice: don't peek at holdout results during development, only at the end.
For product teams
Non-negotiable for credible results. It's the difference between a real benchmark and a cherry-picked demo.
For engineers
Create before any model training. Version control the split. Never use for hyperparameter tuning.
Related
- Foundation of honest evaluation.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome