Noise floor
The minimum detectable change in your metric, below which differences are just noise.
Think of it like
Audio equipment with background hum at 30dB — you can't hear signals quieter than that.
Example
Your eval metric is 85% ± 5% (std 5%). The noise floor is around 5%. Changes smaller than that are invisible in the noise.
How it actually works
Knowing your noise floor is critical. If you can only detect 2% changes, a 1% improvement is meaningless even if it's technically better. Lowering the noise floor requires larger eval sets (expensive) or lower-variance metrics (hard to design). Many papers ignore noise floor and claim credit for improvements smaller than it.
For product teams
Only trust improvements larger than your noise floor. Anything smaller is statistical illusion.
For engineers
Compute from your eval variance. Report it alongside metrics. Flag claimed improvements smaller than it.
Related
- Separates signal from noise.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome