Match Hopper FP8 and Blackwell FP4 tensor cores to their throughput and bandwidth effects on decode

Match pairs·Medium·4.0 · 0·~2 min·Asked atNVIDIA

Attempt it

Hopper FP8 (E4M3 / E5M2) native math

Blackwell FP4 native math

Bottleneck at decode batch 1 on H100 / FP8

Bottleneck at decode batch 1 on B200 / FP4

Practical implication for serving

**Each new tensor-core precision helps decode on two axes at once.** Hopper introduced native FP8 (two formats, E4M3 and E5M2) and Blackwell introduced native FP4. Both moves doubled raw compute throughput at the new precision and halved the bytes per weight in HBM. Decode at small batch is the workload that pins these GPUs. Each decode step reads the entire model's weights from HBM, multiplies them against a tiny activation vector, and writes one new token's worth of state back. The matmul on a single token is trivial; the HBM read of the full weight tensor is the dominant cost. That makes decode HBM-bandwidth bound, not compute bound. So when FP8 halved bytes per weight versus BF16, decode throughput nearly doubled on the same HBM bandwidth budget. When FP4 halved it again on Blackwell, decode throughput nearly doubled again. The FLOPs doublings each generation also happened, but they only start to matter at higher batch sizes where the workload crosses the roofline knee from memory-bound to compute-bound. The practical serving implication is that the same physical GPU board now serves higher concurrent batch on a memory-bound decode workload. Tokens per second per dollar rises generation over generation precisely because both axes improve together. The serving operator gets to run a larger fleet of concurrent users on the same hardware footprint.

**Two axes, asymmetric importance.** Hopper FP8 and Blackwell FP4 each double tensor-core throughput at the new precision and halve the bytes occupied by each weight in HBM. Decode at small batch is bandwidth bound (the per-step bottleneck is reading the full weight tensor from HBM to compute one new token), so the bandwidth halving is the binding improvement and the FLOPs doubling is a longer-batch story. **Bottleneck at H100 / FP8 decode batch 1.** Still HBM bandwidth bound. FP8 weights halved bandwidth pressure versus BF16, which doubled the achievable tokens per second per slot, but the workload is still sitting on the memory slope of the roofline. Adding tensor-core FLOPs would not help; the GPU is starving for bytes, not for math. **Bottleneck at B200 / FP4 decode batch 1.** Bandwidth halves again. The workload moves further along the roofline; on some configurations the knee from memory-bound to compute-bound shifts so much that small-batch decode starts to flirt with being compute-aware. Practically this means the operator can run a larger concurrent batch (more users at once) before hitting the compute roof. **Practical serving implication.** Same physical HBM budget now fits more concurrent KV-cache and more concurrent active weights at lower-precision storage. Tokens per second per dollar rises generation over generation because both bandwidth and compute improve, and decode mostly cares about the first. The right framing is roofline-driven: FP8 to FP4 is primarily a bandwidth optimization that secondarily improves compute headroom, and the binding improvement at small-batch decode is the first half of that sentence.

Quick reference

TL;DR

Hopper FP8 and Blackwell FP4 each double tensor-core throughput and halve weight bandwidth versus the previous precision; decode is bandwidth bound at small batch so the bandwidth halving is the bigger win.

Memory aid

Easy to grasp

Picture a delivery truck running boxes from a warehouse (the GPU's memory) to a packaging line (the GPU's math units). The warehouse-to-line road has a fixed width, so the rate-limiter is how many boxes per second you can push down that road. If you make each box smaller, you fit more boxes per second on the same road and the line can stay busy. FP8 makes each box half the size of the previous generation; FP4 halves it again. Meanwhile the packaging line itself also got faster each generation, but it was rarely the bottleneck for word-by-word decoding anyway. The big practical win each generation is fitting more boxes through the same road, which lets you serve more concurrent users on the same hardware.

Key concepts

FP8 precision FP4 precision Quantization Roofline model

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: two-axis improvement + decode bandwidth bound at small batch + roofline knee + FP8 formats + FP4 microscaling + accuracy calibration caveats.

Real-world examples

Real products, models, and research that use this idea.

vLLM and TensorRT-LLM both ship FP8 (E4M3) weight quantization and FP8 KV cache for H100 deployments, citing the bandwidth halving as the primary speedup driver.
NVIDIA's TensorRT-LLM Blackwell release notes call out NVFP4 inference for Llama 3 / 4 family models with measured 2x decode throughput over Hopper FP8 on equivalent SKUs.
Together AI and Fireworks publish per-precision throughput tables that show roughly 1.8-2x decode tokens/sec from BF16 to FP8 and another 1.6-1.9x from FP8 to FP4 across 70B-class workloads.
DeepSeek published an MXFP4 inference variant of DeepSeek V4 in 2026 that uses microscaling FP4 weights with per-block scaling factors, validating that FP4 deployment is operationally feasible at frontier scale.
MLPerf Inference 2026 results show Blackwell B200 with FP4 weights leading per-GPU throughput rankings on Llama-class decode benchmarks.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is the microscaling format (MXFP4 / NVFP4) and why does it matter for FP4 accuracy?

QHow does FP4 weight quantization interact with FP8 or BF16 activations?

QWhere does the KV cache fit into this picture?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Citing peak FLOPs as the headline number when comparing Hopper FP8 to Blackwell FP4. Decode is HBM bandwidth bound at small batch, so the byte per weight reduction matters more than the FLOPs doubling on most inference workloads.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.