**Each new tensor-core precision helps decode on two axes at once.** Hopper introduced native FP8 (two formats, E4M3 and E5M2) and Blackwell introduced native FP4. Both moves doubled raw compute throughput at the new precision and halved the bytes per weight in HBM.
Decode at small batch is the workload that pins these GPUs. Each decode step reads the entire model's weights from HBM, multiplies them against a tiny activation vector, and writes one new token's worth of state back. The matmul on a single token is trivial; the HBM read of the full weight tensor is the dominant cost. That makes decode HBM-bandwidth bound, not compute bound.
So when FP8 halved bytes per weight versus BF16, decode throughput nearly doubled on the same HBM bandwidth budget. When FP4 halved it again on Blackwell, decode throughput nearly doubled again. The FLOPs doublings each generation also happened, but they only start to matter at higher batch sizes where the workload crosses the roofline knee from memory-bound to compute-bound.
The practical serving implication is that the same physical GPU board now serves higher concurrent batch on a memory-bound decode workload. Tokens per second per dollar rises generation over generation precisely because both axes improve together. The serving operator gets to run a larger fleet of concurrent users on the same hardware footprint.
**Two axes, asymmetric importance.** Hopper FP8 and Blackwell FP4 each double tensor-core throughput at the new precision and halve the bytes occupied by each weight in HBM. Decode at small batch is bandwidth bound (the per-step bottleneck is reading the full weight tensor from HBM to compute one new token), so the bandwidth halving is the binding improvement and the FLOPs doubling is a longer-batch story.
**Bottleneck at H100 / FP8 decode batch 1.** Still HBM bandwidth bound. FP8 weights halved bandwidth pressure versus BF16, which doubled the achievable tokens per second per slot, but the workload is still sitting on the memory slope of the roofline. Adding tensor-core FLOPs would not help; the GPU is starving for bytes, not for math.
**Bottleneck at B200 / FP4 decode batch 1.** Bandwidth halves again. The workload moves further along the roofline; on some configurations the knee from memory-bound to compute-bound shifts so much that small-batch decode starts to flirt with being compute-aware. Practically this means the operator can run a larger concurrent batch (more users at once) before hitting the compute roof.
**Practical serving implication.** Same physical HBM budget now fits more concurrent KV-cache and more concurrent active weights at lower-precision storage. Tokens per second per dollar rises generation over generation because both bandwidth and compute improve, and decode mostly cares about the first.
The right framing is roofline-driven: FP8 to FP4 is primarily a bandwidth optimization that secondarily improves compute headroom, and the binding improvement at small-batch decode is the first half of that sentence.
Quick reference
TL;DR
Hopper FP8 and Blackwell FP4 each double tensor-core throughput and halve weight bandwidth versus the previous precision; decode is bandwidth bound at small batch so the bandwidth halving is the bigger win.
Picture a delivery truck running boxes from a warehouse (the GPU's memory) to a packaging line (the GPU's math units). The warehouse-to-line road has a fixed width, so the rate-limiter is how many boxes per second you can push down that road. If you make each box smaller, you fit more boxes per second on the same road and the line can stay busy. FP8 makes each box half the size of the previous generation; FP4 halves it again. Meanwhile the packaging line itself also got faster each generation, but it was rarely the bottleneck for word-by-word decoding anyway. The big practical win each generation is fitting more boxes through the same road, which lets you serve more concurrent users on the same hardware.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Workloads with strict accuracy requirements that have not been validated at the target precision FP4 in particular needs careful per-block calibration, and uncalibrated drops can degrade attention scores or long-tail token probabilities.
Already-compute-bound workloads at large batch The bandwidth benefit is already saturated; the FLOPs ceiling matters more, and the improvement is closer to the raw FLOPs doubling rather than the full 2x bandwidth-plus-compute.
Models or layers that are known to be precision-sensitive Some final layers, softmax tails, or routing heads need higher precision; mixed-precision serving (keep these in FP16/BF16) is often required.
Real products, models, and research that use this idea.
1vLLM and TensorRT-LLM both ship FP8 (E4M3) weight quantization and FP8 KV cache for H100 deployments, citing the bandwidth halving as the primary speedup driver.
2NVIDIA's TensorRT-LLM Blackwell release notes call out NVFP4 inference for Llama 3 / 4 family models with measured 2x decode throughput over Hopper FP8 on equivalent SKUs.
3Together AI and Fireworks publish per-precision throughput tables that show roughly 1.8-2x decode tokens/sec from BF16 to FP8 and another 1.6-1.9x from FP8 to FP4 across 70B-class workloads.
4DeepSeek published an MXFP4 inference variant of DeepSeek V4 in 2026 that uses microscaling FP4 weights with per-block scaling factors, validating that FP4 deployment is operationally feasible at frontier scale.
5MLPerf Inference 2026 results show Blackwell B200 with FP4 weights leading per-GPU throughput rankings on Llama-class decode benchmarks.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat is the microscaling format (MXFP4 / NVFP4) and why does it matter for FP4 accuracy?
Sign in or go Pro to see the approach for this follow-up.
QHow does FP4 weight quantization interact with FP8 or BF16 activations?
Sign in or go Pro to see the approach for this follow-up.
QWhere does the KV cache fit into this picture?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Citing peak FLOPs as the headline number when comparing Hopper FP8 to Blackwell FP4. Decode is HBM bandwidth bound at small batch, so the byte per weight reduction matters more than the FLOPs doubling on most inference workloads.