Next-token prediction
The entire job of a language model: given some text, guess what comes next.
Think of it like
Finishing your friend’s sentence — do it well enough, billions of times, and it looks like understanding.
Example
Type “The capital of France is” and the model predicts “Paris” as the most likely next token.
How it actually works
During training the model sees oceans of text and repeatedly guesses the next token, getting corrected each time. That single objective, at scale, is enough to produce grammar, facts, and reasoning-shaped behavior. It’s astonishingly simple, and its limits — it predicts likely, not true — come straight from the objective.
For product teams
Remember the model optimizes for “plausible next word,” not “correct answer” — that gap is where hallucination lives.
For engineers
The self-supervised pretraining objective: minimize cross-entropy of the true next token.
Related
- Autoregressive — Done repeatedly, it is autoregressive.
- Hallucination — Where it goes wrong is hallucination.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome