BF16
A 16-bit floating-point format (from Google Brain) that keeps the same range as float32 but with less precision.
Think of it like
A wide-angle lens that captures a larger field of view but at lower pixel density.
Example
Float32 can represent 0.0000001 to 1000000000. BF16 can too, but with fewer distinct values in between.
How it actually works
BF16 is better than FP16 for training stability because it keeps float32's range. This matters for transformer training where gradients span many orders of magnitude. It's becoming standard in recent models (PaLM, Gemini). The downside: less hardware support than FP16 (though Nvidia newer GPUs support it).
For product teams
Emerging standard. Expect more models to use it.
For engineers
Use BF16 for training if your hardware supports it. Performance is similar to FP16 with better stability.
Related
- Improved half-precision.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome