Why does fp16 fine-tuning need dynamic loss scaling but bf16 doesn't?
fp16's 5-bit exponent has a narrow range, so small gradients underflow to zero and need loss scaling. bf16 shares fp32's 8-bit exponent, so they never underflow.
Think of a number format as a ruler with a fixed number of tick marks. fp16 and bf16 both get 16 bits, but they split them differently. fp16 spends more bits on fine ticks near each value and fewer on reach, so it cannot represent very tiny numbers. bf16 spends more bits on reach and fewer on fine ticks, so it can still represent tiny numbers, just less precisely. During training, gradients get very tiny. On fp16's short ruler they fall off the end and become zero, so the model stops learning. Loss scaling temporarily blows the numbers up so they land back on the ruler, then shrinks them again. bf16's long ruler reaches the tiny numbers already, so it needs no such trick.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This question is really a test of whether you understand floating-point layout, not whether you have memorized a framework flag. Both fp16 and bf16 occupy 16 bits, so people assume they are interchangeable. They are not, and the reason a loss scaler appears for one but not the other comes straight from how those 16 bits are partitioned.
A floating-point number has three parts: a sign bit, an exponent field, and a mantissa field. The value is reconstructed as the sign times the mantissa times two raised to the exponent. The exponent determines the dynamic range, meaning how large and how small a magnitude you can represent at all. The mantissa determines precision, meaning how finely you can resolve values within that range. These two concerns are independent, and the whole bf16 versus fp16 story turns on keeping them separate in your head.
fp16 spends 5 bits on the exponent and 10 on the mantissa. bf16 spends 8 bits on the exponent and 7 on the mantissa. That single design choice is the entire answer: bf16 deliberately copies fp32's exponent width, sacrificing mantissa precision to get fp32's range in half the bits. fp16 keeps more precision but pays with a narrow range that gradients fall out of.
Mixed-precision training emerged around 2017 to halve memory and double throughput on tensor cores while keeping fp32-level accuracy. The original recipe targeted fp16 and bundled loss scaling as a mandatory companion. bf16 arrived later from the TPU world and removed the need for that companion. Understanding why is the goal of this deep dive.
The bit layout sets range and precision
A floating-point value reconstructs from its three fields by the rule:
The number of exponent bits decides how far the exponent can swing, which sets the smallest and largest magnitudes you can represent. The number of mantissa bits decides how many significant figures you keep within any given magnitude.
fp16 allocates 1 sign bit, 5 exponent bits, and 10 mantissa bits. bf16 allocates 1 sign bit, 8 exponent bits, and 7 mantissa bits. fp32, for reference, uses 1 sign bit, 8 exponent bits, and 23 mantissa bits.
Notice the deliberate alignment: bf16 copies fp32's exponent width exactly. That is the design intent. bf16 was built to be a drop-in for fp32 in terms of range, truncating only the mantissa. You can convert fp32 to bf16 by simply chopping the low 16 mantissa bits, which makes the formats trivially interconvertible. fp16 was instead built for graphics, where range mattered less and the extra mantissa precision was useful for storing colors and textures. Repurposing it for deep learning is what exposed the underflow problem.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | fp16 (float16) | bf16 (bfloat16) |
|---|---|---|
| Bit layout | 1 sign + 5 exponent + 10 mantissa | 1 sign + 8 exponent + 7 mantissa |
| Dynamic range | Narrow (about 10^-5 to 10^4) | Wide (matches fp32) |
| Mantissa precision | Higher (10 bits) | Lower (7 bits) |
| Loss scaling | Required to avoid gradient underflow | Not needed |
| Typical hardware | Older GPUs, inference kernels | Ampere and newer, TPUs |
Real products, models, and research that use this idea.
- PyTorch torch.cuda.amp uses a GradScaler for fp16 autocast but skips it entirely for bf16 autocast, exactly because bf16 cannot underflow.
- NVIDIA Ampere, Hopper, and Blackwell GPUs added native bf16 tensor cores, which is why frameworks default to bf16 for training on modern datacenter hardware.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does mixed-precision training keep an fp32 master copy of the weights even when computing in bf16?
Think about repeated tiny updates accumulating into a low-precision accumulator. Discuss how a coarse mantissa loses small increments, and why a high-precision master weight plus fp32 optimizer state preserves them.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blaming fp16's mantissa for underflow. The mantissa controls precision, not range. Underflow is an exponent problem, and fp16 has only 5 exponent bits.
60 second bullets to scan on the way to the call.
Bit split of fp16 versus bf16 and what each field controls
Why exponent width sets dynamic range
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.