Where does FP8 fit in modern LLM serving and which GPUs support it?

**FP8** is an 8-bit floating-point number format, half the size of FP16. Halving the bit width has two big effects on LLM serving. First, model weights take half the space in memory. A 70B model in FP16 needs about 140 GB. In FP8 it needs 70 GB and fits on a single 80 GB H100 GPU. HBM bandwidth pressure also halves, because every step reads half as many bytes. Second, tensor cores can multiply FP8 numbers at roughly double the rate of FP16. The same physical multiply unit packs twice as many narrow operands per cycle. On an H100 SXM this raises dense peak throughput from about 990 TFLOP/s in BF16 to about 1,979 TFLOP/s in FP8. There are two FP8 variants. **E4M3** has 4 exponent and 3 mantissa bits. It favours precision and is used for weights and forward-pass activations. **E5M2** has 5 exponent and 2 mantissa bits. It favours dynamic range and is mainly used for gradients during training. Native FP8 tensor cores arrived with NVIDIA Hopper (H100) in 2022 and continue in H200 and Blackwell (B100, B200). Older GPUs such as the A100 lack them, so FP8 math there gets no speedup.

**FP8** is an 8-bit floating-point format that halves memory and HBM bandwidth versus FP16 while roughly doubling tensor-core throughput on matmul-heavy kernels. It is the default low-precision format for production LLM serving. There are two standardised variants. **E4M3** uses 4 exponent bits and 3 mantissa bits, optimised for the precision needs of weights and forward-pass activations. **E5M2** uses 5 exponent bits and 2 mantissa bits, optimised for the wide dynamic range needed by training gradients. Inference typically uses E4M3 for both weights and activations. Training mixed precision uses E5M2 on the backward path. Native hardware support began with NVIDIA Hopper. The H100 launched in 2022 with FP8 tensor cores that multiply FP8 operands and accumulate in FP16 or FP32. The H200 keeps the same FP8 peak with more HBM and bandwidth. Blackwell B200 adds more HBM and a higher FP8 peak. Ampere (A100) and older GPUs have no FP8 tensor cores. So FP8 weights must be converted to 16-bit in software, and the math gets no speedup. Accuracy degradation versus FP16 is modest for most LLMs when using per-tensor or per-channel scales and calibration. Production stacks like TensorRT-LLM, vLLM, and SGLang ship FP8 quantisation recipes that recover almost all the FP16 accuracy on standard benchmarks. For more aggressive savings, FP4 on Blackwell goes another step lower, at higher accuracy cost.