What unit measures GPU memory bandwidth and why does that number cap decode speed?

**Memory bandwidth** is the rate at which data can move between the GPU's main memory (called HBM, high-bandwidth memory) and the compute cores that actually do math. It is measured in gigabytes or terabytes per second. An H100 SXM has about 3.3 TB/s of HBM bandwidth. An A100 has about 2 TB/s. A newer B200 has about 8 TB/s. This number matters because LLM decoding is bottlenecked by reading bytes, not by doing math. Every time the model generates one new token, the GPU has to read all the model's weights out of HBM and into the compute cores. For a 70-billion-parameter model in bf16, that is 140 GB of data per token. The math itself, multiplying that one token's vector against the loaded weights, is trivial in comparison. If an H100 can move 3.3 TB/s of bytes and you need to read 140 GB per token, the theoretical ceiling on decode speed is about 23 tokens per second per request, regardless of how many FLOPs the GPU is capable of. You could double the FLOPs and the number would not budge. That is what people mean when they say decode is bandwidth-bound.

**Memory bandwidth** is the bytes-per-second rate at which data moves between HBM (the GPU's main memory) and the streaming multiprocessors that execute compute. It is measured in GB/s or TB/s, and current-generation flagships sit at roughly 2 TB/s (A100 HBM2e), 3.3 TB/s (H100 SXM HBM3), and 8 TB/s (B200 HBM3e). Decode performance is bandwidth-bound because the arithmetic intensity of a single decode step is extremely low. Each new token's forward pass requires reading the entire model weights once (because you cannot avoid loading the matrices you will multiply against) plus reading the full KV cache (so attention can see prior tokens) plus reading a small activation tensor for the new token. The number of FLOPs done is roughly two FLOPs per parameter read, so the FLOP-per-byte ratio sits near 1, far below the hundreds to thousands ratio modern GPUs need to saturate their tensor cores. The roofline interpretation is direct. With intensity that low, you sit far down on the bandwidth-roof slope, not under the compute-roof flat ceiling. The achievable throughput equals bandwidth times intensity, which is a fixed dollar amount of bytes-per-token. Doubling FLOPs would not help. Halving the bytes-per-token (via quantization, KV cache compression, or speculative decoding) would directly double tokens-per-second. This is why almost every inference optimization people care about, FP8, INT4 weights, paged attention, multi-query attention, is fundamentally an attempt to reduce bytes read per token.