Predict the bandwidth-vs-compute latency of a single Llama-70B decode step on H100

**Decode** generates one token at a time. To produce that single token, the GPU must read every model weight out of HBM, the high-bandwidth memory next to the chip. For a 70B model in FP16 that is about 140 GB of data movement per token. The H100 reads HBM at roughly 3 TB/s. So moving 140 GB takes 140 divided by 3000, which is about 0.047 seconds, or 47 ms. That is the time just to fetch the weights. Now the math. The forward pass for one token is about 140 GFLOPs, and the H100 does 989 TFLOPS in FP16. So compute takes 140 divided by 989000, about 0.00014 seconds, or 0.14 ms. Compare the two. Reading weights takes 47 ms, while doing the math takes 0.14 ms. The memory read is about 300 times slower than the compute. The GPU spends almost all its time waiting on memory, not calculating. This is why batch-1 decode is called bandwidth-bound. The fix is never a faster math unit. The fix is moving fewer bytes, or reusing each weight read across more tokens.

**Decode is memory-bandwidth-bound** at batch 1. The bottleneck is reading the weights out of HBM, not doing the arithmetic. Two divisions prove it. Weight-read time is bytes divided by bandwidth: $$t_\text{mem} = \frac{\text{params} \cdot \text{bytes}}{\text{bandwidth}}$$ Plug in the numbers. That is 140 GB over 3 TB/s, which gives about 47 ms per token. Compute time is FLOPs over peak FLOPS. That is 140 GFLOPs over 989 TFLOPS, which gives about 0.14 ms. The ratio is roughly 335x, about 2.5 orders of magnitude. Bandwidth wins overwhelmingly. The 989 TFLOPS of the chip sits almost entirely idle because data cannot arrive fast enough to feed it. This single number is the foundation of decode optimization. The latency floor is the 47 ms weight read. To go faster you must cut bytes moved, with quantization. Or you amortize the one weight read across many tokens, with batching or speculative decoding. Cutting FLOPs does nothing here. It attacks the term that is already 300 times smaller.