Where does FP8 fit in modern LLM serving and which GPUs support it?

Attempt it

FP8 is an 8-bit floating-point format with two common variants: E4M3 (4 exponent + 3 mantissa bits, used for weights and activations) and E5M2 (used for gradients). It halves memory and HBM bandwidth versus FP16 with only a modest accuracy hit, and roughly doubles tensor-core throughput on matmul-heavy phases. Native FP8 tensor cores landed in NVIDIA Hopper (H100/H200) and continue in Blackwell (B100/B200).

Your answer

Quick reference

TL;DR

FP8 is an 8-bit floating-point format with E4M3 and E5M2 variants; native Hopper tensor cores arrived in 2022 (H100), making FP8 the default low-precision inference format for modern LLMs.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture squeezing each number in a giant spreadsheet from a 16-digit field down to an 8-digit field. The total size of the spreadsheet halves and the chip can do twice as many operations in the same time, because each cell is half as wide. Most calculations still come out almost the same because the model never really needed those extra digits. There are two ways to split the 8 digits between the size part and the precise part. One leans toward precision and is used for the numbers in the trained model. The other leans toward huge ranges and is used during training when some numbers swing wildly. Modern data-center chips have built-in hardware to multiply these tiny numbers at full speed.

Key concepts

FP8 (E4M3 and E5M2)Quantization NVIDIA Hopper architecture Tensor cores

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define FP8 as 8-bit float with E4M3 and E5M2 variants, anchor native Hopper tensor cores in 2022, quantify the 2x throughput and halved memory, walk through quantisation calibration, then connect to FP8 KV cache and Blackwell's FP4 successor.

Real-world examples

Real products, models, and research that use this idea.

NVIDIA H100 SXM ships native FP8 tensor cores at about 1980 TFLOP/s peak, the headline inference number for Hopper.
TensorRT-LLM and vLLM provide FP8 quantisation pipelines for Llama 4, DeepSeek V4, and Qwen 3.5 with under 1 percent benchmark drop.
SGLang ships FP8 KV cache as a default option on H100, raising decode batch ceilings by roughly 2x.
Anthropic and OpenAI run serving fleets primarily in FP8 on Hopper and Blackwell, the basis for current pricing.
DeepSeek V4 training used FP8 mixed precision (E5M2 for gradients, E4M3 for forward) to fit larger batches on H800 clusters.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy use E4M3 for activations and E5M2 for gradients rather than the same format throughout?

Sign in or go Pro to see the approach for this follow-up.

QHow does FP8 KV cache change decode performance, separately from FP8 weights?

Sign in or go Pro to see the approach for this follow-up.

QWhat accuracy techniques recover the FP8 quantisation gap in production?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Calling FP8 a single format when it has two distinct variants. E4M3 leans precision for forward-pass tensors; E5M2 leans dynamic range for gradients. Production stacks use both, not interchangeably.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.