bf16 keeps fp32's 8-bit exponent in 16 bits, so training values rarely overflow or underflow, and modern tensor cores run it as fast as fp16.
Picture two rulers of the same length. The fp16 ruler has fine tick marks, but it only measures up to 65,504. A very small number falls off one end and becomes zero, and a very large number falls off the other end and becomes infinity. The bf16 ruler has coarser tick marks, but it covers the same huge span as the fp32 ruler. Training a large model makes some numbers tiny (gradients) and some numbers large (activation spikes). With the fp16 ruler you must keep sliding numbers into its short window, which is what loss scaling does. With the bf16 ruler they already fit. You read each number less exactly, but the careful sums are done on an fp32 ruler anyway.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
fp16 and bf16 both store a number in 16 bits, so the difference between them is not size. It is how the 16 bits are split. bf16 moves three bits from the mantissa to the exponent, which buys fp32's full range at the cost of precision.
For pretraining, values span many orders of magnitude, and a single inf can ruin a run that has used weeks of cluster time. Range is worth more than precision there. On current accelerators the trade also costs nothing in speed, so bf16 became the default training format.
Same 16 bits, different split
A binary floating-point number has three parts: a sign bit, exponent bits that pick a power of two, and mantissa bits that give the digits inside that power.
The exponent width sets the range. The mantissa width sets the precision, meaning how close two neighboring values are. The three formats in a mixed-precision run split their bits like this:
| Format | Sign | Exponent | Mantissa | Largest value | Smallest normal value | Relative step |
|---|---|---|---|---|---|---|
| fp32 | 1 | 8 | 23 | about 3.4e38 | about 1.2e-38 | about 1.2e-7 |
| fp16 | 1 | 5 | 10 | 65,504 | about 6.1e-5 | about 9.8e-4 |
| bf16 | 1 | 8 | 7 | about 3.4e38 | about 1.2e-38 | about 7.8e-3 |
bf16 is the top 16 bits of an fp32 number. Converting fp32 to bf16 rounds away the lower 16 mantissa bits, and the exponent passes through unchanged. bf16 can therefore hold any value fp32 can hold, only with fewer significant digits: about 2 to 3 decimal digits, against about 3 to 4 for fp16 and about 7 for fp32.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- BLOOM 176B: early 104B experiments in fp16 on V100 GPUs diverged irreversibly; the final run used bf16 mixed precision on A100s and was stable.
- The Llama 3 paper reports its pretraining efficiency as BF16 MFU of 38-43% on H100 GPUs.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does dynamic loss scaling fix gradient underflow in fp16 but not forward-pass overflow?
The scale multiplies the loss before the backward pass, so it shifts gradients only; forward activations are never scaled.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking bf16 saves memory over fp16. Both use 16 bits; bf16 moves three bits from the mantissa to the exponent, so it gains range and loses precision.
60 second bullets to scan on the way to the call.
Bit split of fp32, fp16 and bf16
fp16 maximum value and underflow threshold
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.