FP8 pretraining stays stable through scaffolding: fine grained scaling factors, higher precision for fragile ops and optimizer state, overflow telemetry, and a bf16 matched ablation before full commitment.
Imagine a moving company that discovers it can fit twice as much on each truck by switching to much smaller boxes. Great for speed, but the small boxes have a catch: anything unusually large gets crushed and anything tiny rattles into dust. So the movers add rules. They measure each batch of items and pick padding so things fit snugly (scaling). The genuinely fragile items, glassware and electronics, still ride in big boxes (higher precision). At every stop someone checks for crushed corners (monitoring). And before signing the contract for the whole house, they do one trial run and compare it against the old method (the ablation). Small boxes plus discipline works; small boxes alone destroys the cargo.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every precision drop in deep learning history has followed the same script. fp32 to fp16 promised free speed and delivered divergence until loss scaling was invented. fp16 to bf16 traded mantissa bits for range and made loss scaling unnecessary. Now bf16 to FP8 roughly doubles matmul throughput on Hopper and later GPUs, and once again the headline number is real while the bare format is unusable.
The interview question here is really asking: do you know which scaffolding makes FP8 work, and can you spot proposals that sound technical but invert the actual mechanics? Walking through the four guardrails and the two traps covers most of what a practitioner needs to know about low precision pretraining.
Why FP8 is tempting and why it is fragile
An FP8 number has eight bits total. The E4M3 variant spends four on the exponent and three on the mantissa, giving a maximum representable value of 448 and very coarse spacing between representable numbers. The E5M2 variant trades mantissa for exponent, reaching about 57344 with even coarser precision. Compare bf16, which shares fp32's full exponent range. FP8 is not a slightly smaller bf16; it is a fundamentally narrow window.
The payoff is hardware: Hopper tensor cores execute FP8 matmuls at roughly twice the bf16 rate, and memory traffic for activations halves. For a pretraining run where dense matmuls dominate the FLOP count, that is a budget level saving, the kind that turns a 60 day reservation into a 40 day one.
The fragility follows directly from the window. Activation and gradient distributions in a transformer span many orders of magnitude and drift as training progresses. Values above the format maximum overflow to infinity or saturate; values far below it underflow to zero. Either event, repeated across billions of elements per step, corrupts the optimization signal. So the engineering question is never whether to use FP8 raw. It is how to keep every tensor's live values inside a window that was never designed to hold them.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DeepSeek-V3 trained its 671B parameter MoE with FP8 matmuls using fine grained tile and block scaling, keeping norms, optimizer states, and master weights in higher precision.
- NVIDIA Transformer Engine on H100 and B200 ships delayed scaling with amax history tracking as the standard FP8 workflow for Megatron and NeMo runs.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy do recipes use both E4M3 and E5M2 in one run?
Compare what activations and weights need against what the backward signal needs: precision near zero versus dynamic range.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Casting everything to FP8 and treating it as a free 2x speedup. The wins come only from matmuls; norms, softmax, optimizer state, and master weights must stay in higher precision or the run drifts and diverges.
60 second bullets to scan on the way to the call.
Which two FP8 formats exist and how do their range and precision differ?
Why does FP8 need scaling factors at all?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.