Cross-entropy
The standard scorecard for how far a model’s predicted probabilities are from the true answer.
Think of it like
Like grading a weather forecaster not on right/wrong but on how confident they were in what actually happened.
Example
During training, the model predicts the next token; cross-entropy is high when it put low probability on the word that actually came next, low when it was confident and correct.
How it actually works
Cross-entropy measures the cost of encoding the real data using the model’s predicted distribution — it’s minimized when the two match. It’s the loss almost every language model trains on, it connects directly to perplexity (its exponential), and it rewards putting probability mass on the truth, not just guessing right.
For product teams
The number that goes down as a model learns; the training objective in one word.
For engineers
−Σ y·log p between target and predicted distributions; the per-token training loss whose exp is perplexity.
Related
- Entropy — Its uncertainty baseline.
- Perplexity — Its exponential, for reporting.
- Log probability — What it’s computed from.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome