Why does fp16 fine-tuning need dynamic loss scaling but bf16 doesn't?
fp16's 5-bit exponent has a narrow range, so small gradients underflow to zero and need loss scaling. bf16 shares fp32's 8-bit exponent, so they never underflow.
Think of a number format as a ruler with a fixed number of tick marks. fp16 and bf16 both get 16 bits, but they split them differently. fp16 spends more bits on fine ticks near each value and fewer on reach, so it cannot represent very tiny numbers. bf16 spends more bits on reach and fewer on fine ticks, so it can still represent tiny numbers, just less precisely. During training, gradients get very tiny. On fp16's short ruler they fall off the end and become zero, so the model stops learning. Loss scaling temporarily blows the numbers up so they land back on the ruler, then shrinks them again. bf16's long ruler reaches the tiny numbers already, so it needs no such trick.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: bit layouts of fp16 vs bf16, exponent equals range vs mantissa equals precision, why gradients underflow in fp16, how dynamic loss scaling works, why bf16's fp32 range removes the need, and the precision trade-off.
| Property | fp16 (float16) | bf16 (bfloat16) |
|---|---|---|
| Bit layout | 1 sign + 5 exponent + 10 mantissa | 1 sign + 8 exponent + 7 mantissa |
| Dynamic range | Narrow (about 10^-5 to 10^4) | Wide (matches fp32) |
| Mantissa precision | Higher (10 bits) | Lower (7 bits) |
| Loss scaling | Required to avoid gradient underflow | Not needed |
| Typical hardware | Older GPUs, inference kernels | Ampere and newer, TPUs |
Real products, models, and research that use this idea.
- PyTorch torch.cuda.amp uses a GradScaler for fp16 autocast but skips it entirely for bf16 autocast, exactly because bf16 cannot underflow.
- NVIDIA Ampere, Hopper, and Blackwell GPUs added native bf16 tensor cores, which is why frameworks default to bf16 for training on modern datacenter hardware.
- Open-weight models like Llama 4 and DeepSeek V4 publish bf16 checkpoints, and most LoRA fine-tunes of them run in bf16 with no loss scaler.
- Google TPUs pioneered bfloat16 and have trained large models in it without loss scaling since the original BERT and T5 era.
- Hugging Face Transformers Trainer exposes bf16=True and fp16=True flags, and only the fp16 path instantiates a dynamic loss scaler.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does mixed-precision training keep an fp32 master copy of the weights even when computing in bf16?
QHow does a dynamic loss scaler actually pick its scale factor at runtime?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Blaming fp16's mantissa for underflow. The mantissa controls precision, not range. Underflow is an exponent problem, and fp16 has only 5 exponent bits.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.