Entropy
A measure of how spread out and uncertain a set of probabilities is.
Think of it like
Like a weather forecast: “50% rain” is high-entropy uncertainty, “99% sunny” is low.
Example
When the model’s next-token distribution has high entropy, several words are nearly tied — a sign it’s genuinely unsure what comes next.
How it actually works
Entropy is highest when options are evenly likely and zero when one outcome is certain. In language models it quantifies uncertainty at each step, and it’s the theoretical floor that cross-entropy loss measures against — training pushes the model’s predictions to match reality’s true distribution.
For product teams
A principled way to detect “the model is guessing here” and route those cases to a human.
For engineers
−Σ p·log p over a distribution; maximal for uniform, zero for a point mass, and the baseline cross-entropy compares to.
Related
- Cross-entropy — Training’s loss built on top of it.
- Softmax temperature — Temperature raises or lowers it.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome