Where does FP8 fit in modern LLM serving and which GPUs support it?
FP8 is an 8-bit floating-point format with E4M3 and E5M2 variants; native Hopper tensor cores arrived in 2022 (H100), making FP8 the default low-precision inference format for modern LLMs.
Picture squeezing each number in a giant spreadsheet from a 16-digit field down to an 8-digit field. The total size of the spreadsheet halves and the chip can do twice as many operations in the same time, because each cell is half as wide. Most calculations still come out almost the same because the model never really needed those extra digits. There are two ways to split the 8 digits between the size part and the precise part. One leans toward precision and is used for the numbers in the trained model. The other leans toward huge ranges and is used during training when some numbers swing wildly. Modern data-center chips have built-in hardware to multiply these tiny numbers at full speed.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
FP8 is the default low-precision format for production LLM serving in 2026. It halves memory versus FP16, halves HBM bandwidth pressure during decode, and roughly doubles tensor-core throughput on matmul-heavy kernels. For most serving teams it is the single highest-leverage optimisation they can ship.
Understanding FP8 well means knowing not just the bit layout but how the two variants differ, which GPUs accelerate it, what calibration techniques close the accuracy gap, and how FP8 KV cache compounds with FP8 weights to multiply decode batch ceilings.
This deep dive walks through the bit format, the H100 launch that made FP8 economically viable, the production quantisation recipes that hold accuracy, the operational impact on decode throughput, and the position of FP8 relative to its successor FP4 on Blackwell.
The goal by the end is to be able to reason about FP8 the way a serving engineer does: knowing what it costs in accuracy, what it buys in throughput, where it breaks, and what calibration moves recover the gap. Those conversations come up in every cost-optimisation review for an LLM-serving stack.
Bit layout: E4M3 and E5M2
FP8 follows the same sign exponent mantissa structure as larger floating-point formats, just with 8 total bits to distribute. The standard variants are:
- E4M3: 1 sign bit, 4 exponent bits, 3 mantissa bits. The exponent is biased at 7, giving exponents from -6 to 8. Max representable is around 448, min normal is about 2 to the minus 6. The 3 mantissa bits offer 8 fractions per binade.
- E5M2: 1 sign bit, 5 exponent bits, 2 mantissa bits. The exponent is biased at 15, giving exponents from -14 to 15. Max representable is around 57344, min normal is about 2 to the minus 14. Only 4 fractions per binade.
E4M3 trades dynamic range for precision. It is well suited to weights and forward activations, which sit in bounded ranges after calibration. E5M2 trades precision for range. It is well suited to gradients, which can vary by many orders of magnitude during training and would underflow in E4M3.
Production inference uses E4M3 for both weights and activations. Training mixed precision often uses E4M3 forward and E5M2 backward, with FP32 master weights and an FP32 accumulator. This is what the NVIDIA Transformer Engine implements.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- NVIDIA H100 SXM ships native FP8 tensor cores at about 1980 TFLOP/s peak, the headline inference number for Hopper.
- TensorRT-LLM and vLLM provide FP8 quantisation pipelines for Llama 4, DeepSeek V4, and Qwen 3.5 with under 1 percent benchmark drop.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy use E4M3 for activations and E5M2 for gradients rather than the same format throughout?
Activations and weights cluster near zero with relatively bounded dynamic range; precision matters more than reach, so E4M3 (more mantissa) is better. Gradients during training can swing across many orders of magnitude as updates compound; E5M2 (more exponent) prevents underflow at the cost of mantissa precision. Mixed-precision training picks each variant for the regime it suits.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Calling FP8 a single format when it has two distinct variants. E4M3 leans precision for forward-pass tensors; E5M2 leans dynamic range for gradients. Production stacks use both, not interchangeably.
60 second bullets to scan on the way to the call.
The bit layout of E4M3 (1 sign, 4 exponent, 3 mantissa) and E5M2 (1 sign, 5 exponent, 2 mantissa)
Which variant is used for weights and activations versus gradients
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.