Calibration
When a model's confidence matches its actual accuracy—it says 90% confident on things it's right about 90% of the time.
Think of it like
Like a weather forecast that says 80% rain and it actually rains 80% of the days it predicts that.
Example
The model predicts a label and assigns 95% confidence. If you look at all predictions with 95% confidence, 95% are correct.
How it actually works
Uncalibrated models are dangerous because you trust them more than you should. Neural nets are often overconfident. You can improve calibration by temperature scaling (adjusting softmax temperature after training) or post-hoc calibration on a holdout set. Perfect calibration is impossible if you don't have labeled examples for every prediction.
For product teams
Check model calibration before shipping—an overconfident model that's wrong will erode trust.
For engineers
Measure calibration with calibration plots; apply temperature scaling if miscalibrated.
Related
- Confidence Calibration — Related metric.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome