Match Hopper FP8 and Blackwell FP4 tensor cores to their throughput and bandwidth effects on decode

**Each new tensor-core precision helps decode on two axes at once.** Hopper introduced native FP8 (two formats, E4M3 and E5M2) and Blackwell introduced native FP4. Both moves doubled raw compute throughput at the new precision and halved the bytes per weight in HBM. Decode at small batch is the workload that pins these GPUs. Each decode step reads the entire model's weights from HBM, multiplies them against a tiny activation vector, and writes one new token's worth of state back. The matmul on a single token is trivial; the HBM read of the full weight tensor is the dominant cost. That makes decode HBM-bandwidth bound, not compute bound. So when FP8 halved bytes per weight versus BF16, decode throughput nearly doubled on the same HBM bandwidth budget. When FP4 halved it again on Blackwell, decode throughput nearly doubled again. The FLOPs doublings each generation also happened, but they only start to matter at higher batch sizes where the workload crosses the roofline knee from memory-bound to compute-bound. The practical serving implication is that the same physical GPU board now serves higher concurrent batch on a memory-bound decode workload. Tokens per second per dollar rises generation over generation precisely because both axes improve together. The serving operator gets to run a larger fleet of concurrent users on the same hardware footprint.

**Two axes, asymmetric importance.** Hopper FP8 and Blackwell FP4 each double tensor-core throughput at the new precision and halve the bytes occupied by each weight in HBM. Decode at small batch is bandwidth bound (the per-step bottleneck is reading the full weight tensor from HBM to compute one new token), so the bandwidth halving is the binding improvement and the FLOPs doubling is a longer-batch story. **Bottleneck at H100 / FP8 decode batch 1.** Still HBM bandwidth bound. FP8 weights halved bandwidth pressure versus BF16, which doubled the achievable tokens per second per slot, but the workload is still sitting on the memory slope of the roofline. Adding tensor-core FLOPs would not help; the GPU is starving for bytes, not for math. **Bottleneck at B200 / FP4 decode batch 1.** Bandwidth halves again. The workload moves further along the roofline; on some configurations the knee from memory-bound to compute-bound shifts so much that small-batch decode starts to flirt with being compute-aware. Practically this means the operator can run a larger concurrent batch (more users at once) before hitting the compute roof. **Practical serving implication.** Same physical HBM budget now fits more concurrent KV-cache and more concurrent active weights at lower-precision storage. Tokens per second per dollar rises generation over generation because both bandwidth and compute improve, and decode mostly cares about the first. The right framing is roofline-driven: FP8 to FP4 is primarily a bandwidth optimization that secondarily improves compute headroom, and the binding improvement at small-batch decode is the first half of that sentence.