Tokenizer training
Also called Tokenizer Training
Building the model's vocabulary by finding which character chunks are common enough to deserve their own token.
Think of it like
Deciding which phrases are common enough to earn a shorthand in your notes, so you write less.
Example
A team runs byte-pair encoding over the corpus, merging frequent character pairs until it has a 100k-token vocabulary.
How it actually works
Before a model trains, its tokenizer is fit on a data sample, usually with byte-pair encoding or a similar algorithm that repeatedly merges the most frequent adjacent units. The result is the fixed vocabulary the model will use forever. It matters because it decides how efficiently text is encoded — a tokenizer fit mostly on English will chop other languages into many more tokens, making them costlier and often worse-served.
For product teams
A one-time choice that quietly sets per-language cost and quality for the model's whole life.
For engineers
Fit a subword vocabulary (e.g. BPE) on a corpus sample by iterative frequency-based merges; fixed thereafter.
Related
- Vocabulary — The result it produces.
- Vocabulary size — The size decision it involves.
- Byte-pair encoding — The common algorithm used.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome