Byte-pair encoding
Also called BPE
The recipe most tokenizers use to build a vocabulary by repeatedly merging the most common pairs of symbols.
Think of it like
Like inventing shorthand: you notice “th” appears constantly, so you give it one squiggle, then merge more from there.
Example
Training a tokenizer, BPE starts from raw bytes and merges frequent pairs until “ing” and “tion” become single tokens.
How it actually works
Starting from individual characters or bytes, BPE counts adjacent pairs, merges the most frequent into a new symbol, and repeats thousands of times. The result is a vocabulary where common strings are one token and rare ones split into subwords — a data-driven balance of vocabulary size against sequence length.
For product teams
The unglamorous step that decides your token economics before a model ever runs.
For engineers
Greedy merge algorithm over a corpus that iteratively fuses the highest-frequency adjacent symbol pair into the vocab.
Related
- Subword — The units it produces.
- Tokenization — The broader process it powers.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome