Byte-Level Model
A model that treats text as a raw stream of bytes, so nothing is ever “out of vocabulary.”
Think of it like
Like a scanner that reads any document as pixels — it never trips over a font or symbol it hasn’t seen.
Example
A byte-level model can process a garbled log file full of odd characters without special handling.
How it actually works
Every piece of text is ultimately bytes, so a byte-level model has a tiny fixed alphabet and can represent literally anything — code, emoji, any language. The price is much longer sequences, since a word becomes many bytes, which strains attention and speed. Architectural tricks that group bytes internally are how researchers try to keep it affordable.
For product teams
Maximum robustness to weird input, paid for in sequence length and compute.
For engineers
A model with a byte vocabulary, trading a fixed small alphabet for longer sequences.
Related
- Tokenizer-Free — The broader family it belongs to.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome