Tokenizer
The tool that chops text into tokens before the model sees it — the reason "strawberry" doesn't have three R's according to AI.
Think of it like
A paper shredder that cuts text into specific-sized pieces — the model only sees the pieces, never the full page.
Example
"unbelievable" becomes ["un", "believ", "able"] — three tokens. The model never sees individual letters.
How it actually works
Most modern tokenizers use Byte Pair Encoding (BPE): start with individual bytes, then iteratively merge the most common pairs into larger tokens. Common words become single tokens; rare words get split into pieces.
For product teams
The invisible text splitter that explains why AI miscounts letters and some languages cost more.
For engineers
BPE/SentencePiece converting text to token IDs; vocabulary size and merge rules affect cost and capability.
Related
- Tokens — Creates the tokens the model reads.
- Hallucination — Why models struggle with character-level tasks like counting letters.
- LLM — Different models use different tokenizers, which affects multilingual performance.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome