Explain bf16's advantage over fp16 and what gradient checkpointing trades

This question has two parts that are easy to mix up, so keep them separate. **bf16 versus fp16.** Both are 16-bit floats, so both halve memory and speed up math compared to fp32. The difference is how they split those 16 bits. fp16 uses a 5-bit exponent and 10-bit mantissa, giving fine precision but a narrow range. bf16 uses an 8-bit exponent and 7-bit mantissa, matching fp32's range but with coarser precision. Wide range is what matters for training, because gradients span many orders of magnitude. With fp16, small gradients fall below its range and become zero, so you need a trick called loss scaling to rescue them. bf16's wide range means gradients fit naturally, so you skip that trick. That is why bf16 is the default on modern GPUs. Gradient checkpointing is unrelated to precision. Normally the backward pass needs every activation from the forward pass, which eats memory. Checkpointing saves only a few activations and recomputes the rest. You trade extra compute for much less memory. Turn it on when memory is tight; leave it off when you have memory to spare and want maximum speed.

**bf16** and fp16 are both 16-bit floats, but they allocate bits differently. fp16 has a 5-bit exponent and 10-bit mantissa. bf16 has an 8-bit exponent and 7-bit mantissa. That 8-bit exponent matches fp32's dynamic range, roughly 10^-38 to 10^38. LLM gradients span that whole range. In fp16's narrow window they routinely underflow to zero or overflow to infinity. fp16 therefore needs dynamic loss scaling. You multiply the loss by a constant, run backward, then divide it out before the optimizer step. The scale auto-adjusts when NaNs appear. bf16 skips all of that, so training is simpler and more stable. The cost is fewer mantissa bits and a slightly noisier update, but training matches fp32 dynamics in practice. Gradient checkpointing is a separate memory technique. The backward pass normally caches every forward activation, which scales with batch times sequence times layers times hidden size. Checkpointing stores activations only at segment boundaries and recomputes the rest during backward. The typical trade is about 70 percent less activation memory for about 30 percent more compute. Enable it when memory-bound; skip it when compute-bound with memory to spare.