Pretraining corpus
Also called Pretraining Corpus
The giant pile of mostly-web text (and code, and more) a base model reads to learn language in the first place.
Think of it like
The entire library a student reads growing up — it shapes everything they later know or assume.
Example
A base model's pretraining corpus might be trillions of tokens drawn from web crawls, books, code, and reference sites.
How it actually works
Pretraining is where the model gets its raw knowledge and language ability, and the corpus is its whole world at that stage. Its scale is measured in trillions of tokens, its composition (the data mixture) shapes strengths and biases, and whatever is absent from it the model simply will not know. Later fine-tuning polishes behavior but rarely adds fundamentally new knowledge the corpus lacked.
For product teams
The source of a model's baseline knowledge — and of the blind spots you inherit.
For engineers
The large, mostly self-supervised text corpus consumed during pretraining; defines the model's knowledge base.
Related
- Pretraining — The stage that consumes it.
- Web crawl — A major source within it.
- Data mixture — The proportions across sources.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome