Pretraining
Also called Pre-training
The long, expensive first phase where a model learns language from a giant pile of raw text.
Think of it like
Like a child soaking up a language for years just by hearing it, before anyone teaches a specific job.
Example
A model reads trillions of tokens of web text, books, and code, predicting the next token over and over, until it has a broad feel for language.
How it actually works
Pretraining is self-supervised: the “labels” are just the next tokens, so no human annotation is needed at this scale. It builds the general capabilities and parametric knowledge everything else builds on, and it’s where most of the compute and cost go. Later stages like fine-tuning and RLHF only steer what pretraining created.
For product teams
The multimillion-dollar foundation you rarely redo; you adapt its result rather than repeat it.
For engineers
Self-supervised next-token training on a massive corpus to learn general representations; the compute-dominant phase.
Related
- Next-token prediction — The self-supervised nature of it.
- Base model — The model it produces.
- RLHF — What refines it afterward.
- Parametric knowledge — Where the knowledge ends up.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome