Decoder. plain-English AI glossary

Variance

● Core

How much your metric bounces around when you measure the same thing multiple times.

Think of it like

A scale that says your weight is different every time you step on it — high variance, not trustworthy.

Example

You run your eval set ten times. You get scores: 84%, 86%, 85%, 87%, 84%, 85%, 86%, 84%, 85%, 85%. High variance would be 70%, 95%, 78%, 92%.

How it actually works

Variance comes from model stochasticity (temperature, sampling), eval set size, and randomness in training. High variance evals are noisy — you can't trust small improvements. Reducing variance means using larger eval sets, lowering temperature, or averaging across runs. The tradeoff: low variance costs more compute.

For product teams

If your eval metric bounces around, you can't tell if changes actually help.

For engineers

Measure by running k times. Compute std and CV. If high, use larger sample or average runs.

Part of eval quality assessment.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome