Text generation
A model producing new text one token at a time, each choice conditioned on everything so far.
Think of it like
Like writing a sentence where you commit to one word, then choose the next based on all the words already down.
Example
You type a prompt and watch the reply stream in word by word — that streaming is generation happening token by token.
How it actually works
Generation is autoregressive: the model predicts a probability distribution for the next token, a decoding strategy picks one, it’s appended, and the loop repeats. Everything from temperature to stop sequences is really about shaping or ending this loop. It’s also why longer outputs cost proportionally more time and money.
For product teams
The core act you’re paying for; output length drives both latency and cost linearly.
For engineers
Iterative next-token sampling from the model’s conditional distribution until a stop condition; inherently sequential.
Related
- Autoregressive — The step-by-step nature of it.
- Next-token prediction — The prediction at each step.
- Sampling — How each token is chosen.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome