Explain bf16's advantage over fp16 and what gradient checkpointing trades

Attempt it

Explain (a) why bf16 is preferred over fp16 for LLM fine-tuning, and (b) what gradient checkpointing trades: when it makes sense and when it doesn't.

Free · 2 AI evals / day

Quick reference

TL;DR

bf16 keeps fp32's 8-bit exponent, so gradients never overflow and loss scaling disappears. Gradient checkpointing trades about 30% compute for about 70% activation memory.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a ruler with a fixed number of marks. You can either cover a huge span of sizes with marks far apart, or a tiny span with marks close together. One kind of number tries to measure finely, so it cannot reach the very smallest sizes, and those just round down to nothing. The other kind spreads its marks wide, so it reaches everything and nothing gets lost. Saving memory is a separate trick. Picture doing a long calculation on scratch paper. Instead of keeping every line you ever wrote, you keep only a few checkpoints and rework the missing lines when you need them. That clears your desk a lot, but you do some sums twice, so it takes a little longer.

Key concepts

Mixed-precision training bfloat16 Dynamic loss scaling Gradient checkpointing

Quick scan · 2–4 min

Everything important, quickly.

4 min: bit allocation of fp16 vs bf16 + why range removes loss scaling + the mantissa cost + gradient checkpointing recompute mechanism + the memory and compute trade + memory-bound vs compute-bound decision.

Side-by-side

Property	fp16	bf16
Exponent / mantissa bits	5 / 10	8 / 7
Dynamic range	Narrow (~10^-5 to 10^4)	fp32-equivalent (~10^-38 to 10^38)
Precision	Higher (10 mantissa bits)	Lower (7 mantissa bits)
Loss scaling	Required (dynamic)	Not needed
Training stability on LLMs	NaN-prone without care	Stable by default

Real-world examples

Real products, models, and research that use this idea.

Llama 3.1 and DeepSeek V4 pretraining and fine-tuning pipelines default to bf16 on H100 and TPU hardware, avoiding fp16 loss scaling entirely.
Hugging Face TRL and Axolotl set bf16=True and gradient_checkpointing=True as the standard recipe for LoRA fine-tuning open-weight models.
Unsloth pairs bf16 with gradient checkpointing to fit Llama 3.1 8B QLoRA runs on a single 24GB consumer GPU.
Google TPUs have used bfloat16 as the native training format since 2018, which is where the format originated.
DeepSpeed and PyTorch FSDP expose bf16 mixed precision plus activation checkpointing as first-class config flags in 2026 training stacks.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does bf16 training match fp32 despite having only 7 mantissa bits?

Sign in or go Pro to see the approach for this follow-up.

QHow do you decide the checkpoint segment granularity in gradient checkpointing?

Sign in or go Pro to see the approach for this follow-up.

QWhen would fp8 enter the picture, and what does it change versus bf16?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.