Statistical significance
Whether a difference in metrics is real or just random variation.
Think of it like
Checking whether a coin is unfair by flipping it many times — one heads isn't proof, but 100 heads in 100 flips is.
Example
Model A scores 85% and Model B scores 86% on your test set. Is B really better, or is that just noise? You run a significance test.
How it actually works
Significance tests tell you whether observed differences are likely under the null hypothesis (they're the same). Low p-value means unlikely under null, so you reject it. The catch: significance depends on sample size (large N makes tiny differences significant) and the test you choose. And significance isn't the same as importance — a tiny statistically significant improvement might not matter.
For product teams
Don't ship based on 1% improvements. Demand both significance and practical importance.
For engineers
Use paired t-tests for comparisons on the same examples, Mann-Whitney for unpaired. Report p-values and confidence intervals.
Related
- Rigor in evaluation.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome