Where does FP8 fit in modern LLM serving and which GPUs support it?

**FP8** is an 8-bit floating-point number format, half the size of FP16. Halving the bit width has two big effects on LLM serving. First, model weights take half the space in memory. A 70B model in FP16 needs about 140 GB; in FP8 it needs 70 GB and fits on a single 80 GB H100 GPU. HBM bandwidth pressure also halves, because every step reads half as many bytes. Second, tensor cores can multiply FP8 numbers at roughly double the rate of FP16. The same physical multiply unit packs twice as many narrow operands per cycle. On an H100 this raises peak throughput from about 990 TFLOP/s in BF16 to about 1980 TFLOP/s in FP8. There are two FP8 variants. **E4M3** has 4 exponent and 3 mantissa bits; it favours precision and is used for weights and forward-pass activations. **E5M2** has 5 exponent and 2 mantissa bits; it favours dynamic range and is mainly used for gradients during training. Native FP8 tensor cores arrived with NVIDIA Hopper (H100, H200) in 2022 and continue in Blackwell (B100, B200). On older GPUs FP8 is software-emulated and offers no speed gain.

**FP8** is an 8-bit floating-point format that halves memory and HBM bandwidth versus FP16 while roughly doubling tensor-core throughput on matmul-heavy kernels. It has become the default low-precision format for production LLM serving in 2026. There are two standardised variants. **E4M3** uses 4 exponent bits and 3 mantissa bits, optimised for the precision needs of weights and forward-pass activations. **E5M2** uses 5 exponent bits and 2 mantissa bits, optimised for the wide dynamic range needed by training gradients. Inference typically uses E4M3 for both weights and activations; training mixed precision uses E5M2 on the backward path. Native hardware support began with NVIDIA Hopper. The H100 launched in 2022 with FP8 tensor cores that multiply FP8 operands and accumulate in FP16 or FP32. The H200 and Blackwell B100/B200 extend this with more HBM and higher FP8 peak rates. On Ampere (A100) and older, FP8 is software-emulated, which is slow and pointless for serving. Accuracy degradation versus FP16 is modest for most LLMs when using per-tensor or per-channel scales and calibration. Production stacks like TensorRT-LLM, vLLM, and SGLang ship FP8 quantisation recipes that recover almost all the FP16 accuracy on standard benchmarks. For more aggressive savings, FP4 on Blackwell goes another step lower, at higher accuracy cost.