Sampling
How the model actually chooses each next word from its list of options — roll the weighted dice, don’t always grab the top one.
Think of it like
Drawing from a bag of tiles where common words have more copies, so surprises are possible but rare.
Example
Ask the same question twice with sampling on and you can get two different-but-valid answers.
How it actually works
After softmax gives a probability per token, sampling picks one at random weighted by those probabilities. That randomness is why models feel creative and non-repetitive — and why they’re not perfectly reproducible unless you fix the seed or switch to greedy decoding.
For product teams
The reason outputs vary run to run; turn it down for consistency, up for variety.
For engineers
Stochastic token selection from the output distribution, controlled by temperature/top-p/top-k.
Related
- Softmax — Chooses among the softmax probabilities.
- Temperature — Tuned by temperature, top-p, and top-k.
- Greedy decoding — Its opposite is greedy decoding.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome