Data mixture
Also called Training Data Mixture
The recipe of how much of each kind of data — web, code, books, math — goes into training, and in what proportion.
Think of it like
A coffee blend: the ratio of beans changes the whole cup, even with the same total amount.
Example
A team bumps the share of code and math in the mixture and the model gets noticeably better at reasoning, at a slight cost to prose.
How it actually works
Pretraining data comes from many sources with very different value, and their proportions strongly shape the final model. Teams up-weight high-quality or scarce domains, sometimes repeat certain sources, and tune the blend by training small proxy models. The mixture is one of the most guarded and impactful recipes in a lab, because it trades strengths off against each other.
For product teams
A hidden dial that decides what your model is naturally good and bad at.
For engineers
The sampling proportions across data sources during pretraining; tuned via proxy runs and domain weighting.
Related
- Pretraining corpus — The full pool the mixture is drawn from.
- Data curation — How sources are cleaned before mixing.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome