bf16 keeps fp32's 8-bit exponent so it matches fp32's dynamic range with less precision. fp16 has only a 5-bit exponent and overflows on values fp32 and bf16 handle fine.
Picture a 16-character licence plate. You can spend characters on the year of the car or on the unique serial. fp32 has 32 characters and lots of room for both. bf16 and fp16 only get 16 characters and have to choose. bf16 spends the same number on year as fp32 does, so it can stamp any year fp32 can, but it has fewer characters left for the serial so two plates can look the same. fp16 spends fewer characters on year and more on serial, so it tells two plates apart more easily but cannot stamp very old or very future years at all. In numerical training that 'year' is dynamic range and 'serial' is precision. bf16 trades precision to keep fp32's range. fp16 keeps more precision but loses range, which is why it needs extra scaffolding to train without overflowing.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
bf16 and fp16 look interchangeable on the surface. Both are 16-bit floating-point formats. Both are widely supported on modern accelerators. Both are used in mixed-precision training to roughly halve memory and double throughput compared to fp32. The interview question of which one to use, and why, separates engineers who have actually trained a model in mixed precision from those who have only read about it.
The answer comes down to how the 16 bits are allocated. fp32 has 1 sign bit, 8 exponent bits, and 23 mantissa bits. The 8-bit exponent gives fp32 its famous dynamic range of roughly 1e-38 to 1e+38, which is enough to handle any gradient or activation that arises in standard neural network training without thinking about overflow. The 23-bit mantissa gives about 7 decimal digits of precision per number, which is more than enough for most computations.
When you compress to 16 bits, you have to give up some bits somewhere. The question is which ones. bf16, the format Google designed for TPU and later adopted everywhere, keeps the 8-bit exponent and chops the mantissa down to 7 bits. The result preserves fp32's dynamic range while sacrificing precision. fp16, the older format defined by the IEEE standard, keeps 10 mantissa bits and uses only 5 exponent bits. The result preserves more precision than bf16 but narrows the dynamic range dramatically, capping the maximum representable value around 6.5e+4 and the minimum around 6e-8.
This layout difference is what drives every practical training decision around precision. bf16 just works for transformer training because the range is fp32-equivalent. fp16 needs loss scaling because gradients during training routinely fall outside its range. This deep dive walks through the bit layouts in detail, explains why range matters more than precision for training, covers the loss-scaling workaround that makes fp16 viable, and ends with the hardware and recipe choices that follow.
Bit layout: 1-8-23 versus 1-8-7 versus 1-5-10
Every IEEE-style floating-point format budgets its bits across three fields: 1 sign bit, some exponent bits, some mantissa bits. The total width is fixed by the format; the split between exponent and mantissa is the design choice.
fp32 uses 1 sign, 8 exponent, 23 mantissa. The exponent encodes powers of 2 from roughly -126 to +127, giving dynamic range from about 1e-38 to about 3.4e+38. The mantissa encodes the significant digits within each exponent slot, giving about 7 decimal digits of precision.
bf16 keeps the 8-bit exponent and truncates the mantissa to 7 bits. Total width is 16 bits. Dynamic range is identical to fp32 because the exponent layout is identical. Precision drops to about 2 to 3 decimal digits per number. The format is literally fp32 with the bottom 16 bits of the mantissa dropped, which is why bf16-to-fp32 conversion is just a zero-padding operation.
fp16 uses 1 sign, 5 exponent, 10 mantissa. The 5-bit exponent encodes powers of 2 from about -14 to +15, giving dynamic range from about 6e-8 to about 6.5e+4. Precision is about 3 to 4 decimal digits per number. The format predates bf16 and was designed for graphics and inference, not for training transformers with huge dynamic ranges of gradients.
The contrast is sharp. bf16 and fp32 share an exponent layout and differ only in mantissa width. bf16 and fp16 share a total width and differ in how they split the budget. The right choice between bf16 and fp16 depends entirely on whether your computation needs more dynamic range or more precision.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | fp32 | bf16 | fp16 |
|---|---|---|---|
| Bit layout (sign / exp / mantissa) | 1-8-23 | 1-8-7 | 1-5-10 |
| Dynamic range (approx) | 1e-38 to 1e+38 | 1e-38 to 1e+38 | 6e-8 to 6.5e+4 |
| Decimal precision | ~7 digits | ~2-3 digits | ~3-4 digits |
| Loss scaling needed for training | No | No | Yes |
| Hardware support | Universal | Ampere+, TPU v3+ | Pascal and later |
Real products, models, and research that use this idea.
- Modern LLM fine-tuning recipes for Llama 4, Qwen 3.5, and DeepSeek V4 default to bf16 mixed precision on H100s and A100s because the dynamic range eliminates loss scaling.
- Hugging Face Transformers' Trainer accepts bf16=True and fp16=True flags; the bf16 flag is preferred when the hardware supports it, the fp16 flag triggers the loss-scaler scaffolding.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does bf16's smaller mantissa not cause training problems for transformers?
Transformer training is more bounded by dynamic range than by precision. Adam optimizer state stays in fp32, master weights stay in fp32, and gradient updates accumulate small per-step changes that bf16's 7-bit mantissa can represent with enough fidelity over many steps. The precision floor matters less when the integration is over many steps.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating bf16 and fp16 as interchangeable. They use different bit budgets, behave differently under gradient updates, and need different training scaffolding.
60 second bullets to scan on the way to the call.
Bit layout of fp32: 1 sign, 8 exponent, 23 mantissa
Bit layout of bf16: 1, 8, 7
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.