Tokens per parameter
Also called Token-to-Parameter Ratio
How many training tokens each model parameter gets to learn from — a rough gauge of whether a model is well-fed or starved.
Think of it like
Students per teacher: too many and each gets too little attention; the ratio tells you if the class is well-served.
Example
A 7B model trained on 2 trillion tokens sees about 285 tokens per parameter — far past the Chinchilla point of 20, on purpose.
How it actually works
Divide training tokens by parameter count. Chinchilla put the compute-optimal ratio near 20, but modern small models are trained to hundreds of tokens per parameter because over-training a small model buys cheaper inference. A low ratio suggests an under-trained, over-sized model; a very high one means you are spending extra training compute to save at serving time.
For product teams
A quick sniff test for whether a model is right-sized for its data budget.
For engineers
Training tokens ÷ parameters; ~20 is compute-optimal, higher trades train cost for inference savings.
Related
- Chinchilla scaling — The study that fixed the optimal value near 20.
- Compute-optimal — The trade-off this ratio expresses.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome