Vocabulary
The fixed set of tokens a model knows — every piece it can read or write has to come from this list.
Think of it like
The full set of LEGO brick shapes in the box; anything you build uses only those.
Example
A common English word is one token in the vocab; a rare name gets spelled out from several.
How it actually works
A tokenizer’s vocabulary is usually tens of thousands of fragments, chosen so frequent text compresses into few tokens. Anything outside it is built up from smaller pieces, which is why unusual words, emoji, or other languages can be surprisingly token-expensive — they don’t have their own dedicated entries.
For product teams
Explains why some inputs cost more tokens than their length suggests.
For engineers
The finite token set from the tokenizer; out-of-vocab text is composed from sub-word units.
Related
- Tokenization — Defined by the tokenization step.
- Tokens — Each entry is a token.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome