Why does fp16 fine-tuning need dynamic loss scaling but bf16 doesn't?
fp16's 5-bit exponent has a narrow range, so small gradients underflow to zero and need loss scaling. bf16 shares fp32's 8-bit exponent, so they never underflow.
Think of a number format as a ruler with a fixed number of tick marks. fp16 and bf16 both get 16 bits, but they split them differently. fp16 spends more bits on fine ticks near each value and fewer on reach, so it cannot represent very tiny numbers. bf16 spends more bits on reach and fewer on fine ticks, so it can still represent tiny numbers, just less precisely. During training, gradients get very tiny. On fp16's short ruler they fall off the end and become zero, so the model stops learning. Loss scaling temporarily blows the numbers up so they land back on the ruler, then shrinks them again. bf16's long ruler reaches the tiny numbers already, so it needs no such trick.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The difference comes from floating-point layout, not from a framework flag. Both fp16 and bf16 occupy 16 bits, so people assume they are interchangeable. They are not. A loss scaler appears for one but not the other because of how each format splits those 16 bits.
A floating-point number has three parts: a sign bit, an exponent field, and a mantissa field. The value is reconstructed as the sign times the mantissa times two raised to the exponent. The exponent determines the dynamic range, meaning how large and how small a magnitude you can represent at all. The mantissa determines precision, meaning how finely you can resolve values within that range. These two concerns are independent, and the bf16 versus fp16 difference turns on keeping them separate.
fp16 spends 5 bits on the exponent and 10 on the mantissa. bf16 spends 8 bits on the exponent and 7 on the mantissa. That single design choice is the whole difference. bf16 deliberately copies fp32's exponent width. It gives up mantissa precision to get fp32's range in half the bits. fp16 keeps more precision but pays with a narrow range that gradients fall out of.
Mixed-precision training emerged around 2017 to halve memory and double throughput on tensor cores while keeping fp32-level accuracy. The original recipe targeted fp16 and bundled loss scaling as a mandatory companion. bf16 arrived with TPU v2 and removed the need for that companion.
The bit layout sets range and precision
A floating-point value reconstructs from its three fields by the rule:
The number of exponent bits decides how far the exponent can swing, which sets the smallest and largest magnitudes you can represent. The number of mantissa bits decides how many significant figures you keep within any given magnitude.
fp16 allocates 1 sign bit, 5 exponent bits, and 10 mantissa bits. bf16 allocates 1 sign bit, 8 exponent bits, and 7 mantissa bits. fp32, for reference, uses 1 sign bit, 8 exponent bits, and 23 mantissa bits.
The alignment is deliberate: bf16 copies fp32's exponent width exactly. bf16 was built to be a drop-in for fp32 in terms of range, truncating only the mantissa. You can convert fp32 to bf16 by chopping the low 16 mantissa bits, so conversion between the formats is trivial. fp16 was instead built for graphics, where range mattered less and the extra mantissa precision was useful for storing colors and textures. Using it for deep learning exposed the underflow problem.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | fp16 (float16) | bf16 (bfloat16) |
|---|---|---|
| Bit layout | 1 sign + 5 exponent + 10 mantissa | 1 sign + 8 exponent + 7 mantissa |
| Dynamic range | Narrow (normal values about 6e-5 to 65504) | Wide (matches fp32) |
| Mantissa precision | Higher (10 bits) | Lower (7 bits) |
| Loss scaling | Required to avoid gradient underflow | Not needed |
| Typical hardware | Older GPUs, inference kernels | Ampere and newer, TPUs |
Real products, models, and research that use this idea.
- PyTorch AMP pairs torch.amp.GradScaler("cuda") with fp16 autocast, while bf16 autocast needs no scaler because bf16 has fp32's range.
- NVIDIA Ampere, Hopper, and Blackwell GPUs added native bf16 tensor cores, which is why frameworks default to bf16 for training on modern datacenter hardware.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does mixed-precision training keep an fp32 master copy of the weights even when computing in bf16?
Think about repeated tiny updates accumulating into a low-precision accumulator. Discuss how a coarse mantissa loses small increments, and why a high-precision master weight plus fp32 optimizer state preserves them.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blaming fp16's mantissa for underflow. The mantissa controls precision, not range. Underflow is an exponent problem, and fp16 has only 5 exponent bits.
60 second bullets to scan on the way to the call.
Bit split of fp16 versus bf16 and what each field controls
Why exponent width sets dynamic range
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.