Confidence Calibration
Ensuring a model's confidence scores are reliable—a 70% confidence prediction is right 70% of the time, not 80%.
Think of it like
Like a weather forecast being honest: if it says 70% rain, it should rain 7 out of 10 times it makes that prediction, not more.
Example
You collect predictions where the model says 80% confident. You check: they're actually right 75% of the time, not 80%. Model is overconfident.
How it actually works
Modern neural nets are notoriously overconfident. Temperature scaling adjusts the softmax temperature to improve calibration. Ensemble methods help (averaging predictions from multiple models reduces overconfidence). Calibration matters for risk-aware decisions (medicine, finance).
For product teams
For high-stakes decisions, measure calibration. A model that's wrong but confident is worse than one that's uncertain.
For engineers
Compute calibration error; apply post-hoc scaling if needed; use ensembles for critical predictions.
Related
- Calibration — Related concept.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome