Subword
A chunk of a word — the middle ground between whole words and single letters that models actually read.
Think of it like
Like Lego bricks for language: “unhappiness” snaps together from “un,” “happi,” and “ness.”
Example
A tokenizer splits “tokenization” into “token” and “ization,” so a rare word still reuses pieces the model already knows.
How it actually works
Whole-word vocabularies balloon and choke on typos and new words; character-level is tiny but makes sequences very long. Subwords are the compromise — common words stay whole, rare ones fragment into familiar parts — which is how models handle words they’ve never literally seen.
For product teams
Why token counts don’t match word counts, and why odd or foreign words cost more.
For engineers
Vocabulary units between characters and words, learned so frequent strings stay intact and rare ones decompose.
Related
- Byte-pair encoding — The algorithm that finds these units.
- Tokenization — The process of splitting text this way.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome