Explain (a) why bf16 is preferred over fp16 for LLM fine-tuning, and (b) what gradient checkpointing trades: when it makes sense and when it doesn't.
bf16 keeps fp32's 8-bit exponent, so small gradients do not underflow and loss scaling disappears. Gradient checkpointing trades about 30% compute for about 70% activation memory.
Imagine a ruler with a fixed number of marks. You can either cover a huge span of sizes with marks far apart, or a tiny span with marks close together. One kind of number measures finely, so it cannot reach the very smallest sizes. Those round down to nothing. The other kind spreads its marks wide, so it reaches everything and nothing gets lost. Saving memory is a separate trick. Picture doing a long calculation on scratch paper. Instead of keeping every line you ever wrote, you keep only a few checkpoints and rework the missing lines when you need them. That clears your desk a lot, but you do some sums twice, so it takes a little longer.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Mixed precision and gradient checkpointing share a goal, fitting a big model on the hardware you have, but solve different problems. Mixed precision picks a cheaper number format. Gradient checkpointing changes what you keep in memory during the backward pass. A common confusion is to call checkpointing a precision trick or to claim bf16 is more accurate.
Both choices are everyday decisions in any real fine-tuning run. You set the precision flag and the checkpointing flag in the first few lines of every training config. Knowing exactly what each one buys, and what it costs, is the difference between a recipe you copied and a recipe you understand.
A clean way to keep them apart is to ask what resource each one is buying back. Mixed precision buys back both memory and math throughput by using a smaller number format. The bf16 versus fp16 choice is about which 16-bit format keeps training stable. Gradient checkpointing buys back memory alone, and it pays for that memory with extra compute. They can be used together, and in practice usually are.
How fp16 and bf16 split their bits
A floating point number splits its bits between an exponent, which sets the magnitude or range, and a mantissa, which sets the precision within that range. Both fp16 and bf16 are 16-bit formats, so each has a fixed budget to divide.
fp16 spends 5 bits on the exponent and 10 on the mantissa, plus 1 sign bit. That buys fine precision but a narrow normal range of roughly 6e-5 to 65504. bf16 instead spends 8 bits on the exponent and 7 on the mantissa. Those 8 exponent bits are exactly what fp32 uses, so bf16 inherits the fp32 range of roughly 10^-38 to 10^38 while giving up precision.
fp16 is more precise but cramped in range. bf16 is coarser but matches the fp32 range. For gradients, range is the property that decides whether training is stable, which is why the bit split matters so much.
fp16 is an IEEE format designed for graphics and inference, where values cluster in a predictable band and precision is at a premium. bf16 was designed at Google specifically for deep learning. There the priority is representing a huge spread of magnitudes rather than resolving them finely. The two formats are answers to two different questions, and LLM training asks the question bf16 was built for. A useful mental shortcut: bf16 is fp32 with the bottom 16 mantissa bits chopped off. Converting between fp32 and bf16 is therefore almost free and never changes the exponent.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | fp16 | bf16 |
|---|---|---|
| Exponent / mantissa bits | 5 / 10 | 8 / 7 |
| Dynamic range | Narrow (normal values ~6e-5 to 65504) | fp32-equivalent (~10^-38 to 10^38) |
| Precision | Higher (10 mantissa bits) | Lower (7 mantissa bits) |
| Loss scaling | Required (dynamic) | Not needed |
| Training stability on LLMs | NaN-prone without care | Stable by default |
Real products, models, and research that use this idea.
- Most open-weight LLM training and fine-tuning pipelines default to bf16 on H100 and TPU hardware, avoiding fp16 loss scaling entirely.
- Hugging Face TRL and Axolotl set bf16=True and gradient_checkpointing=True as the usual method for LoRA fine-tuning open-weight models.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does bf16 training match fp32 despite having only 7 mantissa bits?
Think about where precision is actually needed. Optimizer master weights and accumulations stay in fp32, so rounding noise on the bf16 forward and backward averages out across many steps rather than compounding.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying bf16 is more accurate than fp16. It is actually less precise (7 mantissa bits) but has far wider range. That range is what removes loss scaling.
60 second bullets to scan on the way to the call.
How fp16 and bf16 split their 16 bits differently
Why exponent range, not precision, matters for gradients
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.