Modern training uses **mixed precision**: most tensors in FP16 or BF16 so you can fit more in memory and use tensor cores, but a few numerically sensitive operations stay in FP32 for stability. Attention has exactly one operation that demands FP32 treatment: the softmax.
Matmuls in attention (Q/K/V projections, `QK^T`, the multiplication by V) run fine in FP16/BF16. Tensor cores hit peak throughput in those formats, and matmul is a sum of many products, so per element rounding mostly cancels across the reduction.
Softmax is the exception. The exp function amplifies its input exponentially. In FP16, the largest representable number is around 65504; `exp(11.1)` is already at that ceiling and `exp(12)` overflows to `inf`. Pre-softmax scores after `1/√d` scaling routinely sit in the 5-to-15 range, so exp can easily overflow, produce `inf`, and propagate `NaN` through training.
The fix has two parts: subtract the max before exponentiating (every exp input becomes `<= 0`, output bounded in `(0, 1]`), and accumulate in FP32 so small probabilities don't quantize away. FlashAttention runs an online softmax with FP32 running max and denominator across tiles.
Matmuls in attention, Q/K/V projections, `QK^T`, and the multiplication by V, run in FP16 or BF16 to use tensor cores, which deliver their peak throughput in low precision. The softmax is the exception. It stays in FP32, or uses an online-softmax stabilization, because exp overflows FP16's ~65504 ceiling.
The arithmetic is concrete: FP16 attention scores past about 11 produce `inf`, and once `NaN` shows up it propagates through the V matmul and corrupts training silently. The subtract the max trick bounds exp inputs in `(-inf, 0]` and is essential, but you still want FP32 accumulators for the running max and denominator to preserve the dynamic range of small probabilities.
BF16 helps with the overflow side because its 8-bit exponent matches FP32's range, but the 7-bit mantissa is too coarse for the denominator sum, so FP32 accumulators remain standard even in BF16 stacks. This is not a model-size issue, it's a property of exp.
Quick reference
TL;DR
Matmuls run in FP16/BF16 to use tensor cores; softmax stays in FP32 (or uses running-max stabilization) because exp overflows FP16's ~65504 ceiling.
Imagine a tiny notebook that can only write numbers up to about 65,000. Most of the math in attention is fine, because you're adding lots of small numbers and the result stays in range. But one step asks you to write down e raised to a fairly big number. e to the 11 is already around 60,000. e to the 12 doesn't fit at all. So that one step either needs a bigger notebook with more room, or a clever trick where you shift every number down by the largest one before you write it. Picture using a regular pad for most work and pulling out a giant whiteboard only for that one tricky step.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tiny experimental models on CPU (no tensor cores available) Pure FP32 is faster on CPU; mixed precision only pays off when tensor cores or memory pressure matter.
Inference of already quantized INT8/INT4 weights Quantized inference uses integer arithmetic with its own descaling; the FP16/FP32 mixed precision recipe doesn't apply directly.
Debugging a NaN training run before checking the math Switch to pure FP32 first to isolate whether the bug is precision-related or a real modeling issue, then re-enable mixed precision.
FP16 softmax without max subtraction or FP32 accumulators Scores past ~11 overflow, exp returns inf, NaN propagates through the @V matmul, and training silently corrupts.
Quick scan · 2–4 min
Everything important, quickly.
Walk through which ops live in which precision, give the FP16 overflow numerics for exp, distinguish BF16 vs FP16, describe FlashAttention's online softmax with FP32 running stats, and note FP8 attention on H100 as the active frontier.
Side-by-side
Operation
Typical precision
Why
Q,K,V projections
FP16/BF16 (FP8 on H100)
Tensor core throughput; robust to element-wise error