Scaling laws
Also called Neural Scaling Laws
The predictable way model quality improves as you add more data, parameters, and compute — smooth enough to plan around.
Think of it like
Knowing that doubling the ingredients and oven time reliably yields a bigger, better-baked loaf — the recipe scales.
Example
Before committing to a huge run, a lab uses scaling-law curves fit on small models to predict the loss the big one will hit.
How it actually works
Empirically, test loss falls as a power law in model size, dataset size, and compute — straight lines on a log-log plot across many orders of magnitude. This let labs forecast returns and justify spending on ever-larger runs. The laws describe loss, not capability, and they do not promise where useful behaviors emerge, so they guide budgets more than they guarantee outcomes.
For product teams
The reason "just make it bigger" became a credible strategy — the payoff was forecastable.
For engineers
Power-law relationships between test loss and compute/params/data used to extrapolate and allocate budgets.
Related
- Chinchilla scaling — Uses scaling laws to balance model and data size.
- Compute-optimal — The compute-efficient point on the curve.
- Emergence — Capabilities scaling laws do not directly predict.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome