Decoder. plain-English AI glossary

Tokenization

● Core

Chopping text into the bite-size pieces a model actually reads — usually chunks of words, not whole words.

Think of it like

Slicing a sentence into LEGO bricks before the model can build with it.

Example

“tokenization” might split into token-ization — two pieces, not one word.

How it actually works

A tokenizer learns a fixed vocabulary of common fragments (often via byte-pair encoding), then greedily splits any text into those fragments. It runs before the model sees anything, which is why a model never really sees letters — only the chunks — and why the same sentence can cost more tokens in one language than another.

For product teams

The step that decides your token count — and therefore your cost and length limits.

For engineers

Deterministic sub-word segmentation (BPE/Unigram) mapping text to integer IDs over a fixed vocabulary.

Tokens — The pieces it produces are tokens.
Tokens — Why models miscount letters and fumble rare words.

Read anything AI without the jargon

Look up any term in plain English, or save terms as you read with the free Chrome extension.

Open Decoder Add to Chrome