Walk through the byte accounting that proves a single-batch decode step is bandwidth-bound on H100.

**Decode** is the phase where the model generates output tokens one at a time. To produce a single token, the GPU reads every weight in the model out of high-bandwidth memory, does a little arithmetic, and emits one token. At batch 1 the arithmetic is tiny. Each weight multiplies a single activation value, so the per-token math is a matrix-vector product, not a fat matrix-matrix product. The chip's thousands of math units sit mostly idle while the memory bus does the heavy lifting. That makes decode memory-bandwidth-bound. The time per step is roughly the bytes you must read divided by the memory bandwidth. For Llama-70B in FP16 that is about 140 GB streamed at 3 TB per second, near 47 ms per token. The arithmetic for that same step finishes in under a millisecond. So the GPU spends nearly all of its time waiting on memory, not computing. The practical takeaway is that faster tensor cores do not help single-stream decode. You lower the floor by reading fewer bytes through quantization, reusing one read across many requests through batching, or spreading the bytes over more memory channels through tensor parallelism.

**Decode at batch 1 is bandwidth bound**, and two divisions prove it. First the bytes. Llama-70B has 70 billion parameters. In FP16 each is 2 bytes, so one decode step reads 140 GB of weights from HBM. The KV cache adds a few GB at short context, small enough to ignore for the headline argument. Now the times. The headline relation is: $$t \approx \frac{\text{model bytes}}{\text{bandwidth}}$$ Dividing 140 GB by the 3 TB/s of H100 gives about 47 ms. The compute is a matrix-vector product, roughly 2N or 140 GFLOPs per token. At 989 TFLOPS that finishes in about 0.14 ms. Bandwidth therefore dominates compute by over 300 times. The conclusion is operational. The 47 ms is a latency floor that no kernel trick beats. Three levers move it. Quantization to INT4 cuts the bytes fourfold to roughly 12 ms. Batching shares one read across B requests for B tokens. Tensor parallelism splits the bytes across more parallel HBM. FLOP-cutting tricks touch only the 0.14 ms term and do nothing.