Chinchilla scaling
Also called Chinchilla-Optimal Scaling
The finding that for a given compute budget, models should be smaller and trained on far more data than people assumed.
Think of it like
Realizing a smaller pot on the stove longer cooks better than a giant pot rushed — more time with the heat beats sheer size.
Example
Chinchilla matched a much larger model using fewer parameters but roughly four times the training tokens, at the same compute.
How it actually works
DeepMind's 2022 study showed the compute-optimal ratio is roughly 20 training tokens per parameter — meaning earlier giant models were under-trained. It shifted the field toward smaller, data-hungrier models, which also happen to be cheaper to run. In practice teams now often train well past the Chinchilla point because inference savings justify over-training a smaller model.
For product teams
It quietly rewrote sizing decisions industry-wide and made strong small models the norm.
For engineers
Empirical result: ~20 tokens/param is compute-optimal; smaller models on more data beat larger under-trained ones.
Related
- Compute-optimal — The general principle it is the flagship result of.
- Tokens per parameter — The ratio it pinned down.
- Scaling laws — The curves it sits on.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome