Mixed precision
Also called Mixed-Precision Training
Do most of the training math in low-precision numbers to go faster and use less memory, keeping full precision only where it matters.
Think of it like
Rounding to the nearest cent while you shop, but keeping the exact figures for the final tax return.
Example
A team switches a training run to bf16 and suddenly the same GPUs fit a bigger batch and finish an epoch in half the time.
How it actually works
Weights and activations are stored and multiplied in 16-bit floats (fp16 or bf16) instead of 32-bit, which roughly halves memory and speeds up the matrix math on modern accelerators. The catch is numerical stability — tiny gradients can underflow to zero — so a full-precision master copy of the weights and loss scaling are kept to stop the training from quietly diverging.
For product teams
It is most of why training the same model got cheaper and faster over the last few years, without changing the recipe.
For engineers
Store/compute in fp16 or bf16 with an fp32 master weight copy and dynamic loss scaling to avoid gradient underflow.
Related
- Checkpointing — Pairs with this to fit bigger models in memory.
- Weights — The numbers being rounded are the model weights.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome