Why is HBM bandwidth the binding constraint for LLM decode and how do you reason about the 'memory wall'?

Modern GPUs have two key speeds: how fast they do math (FLOP/s) and how fast they read data from memory (bytes/s). The **memory wall** is the fact that math speed has grown far faster than memory speed across GPU generations. LLM decoding generates one token at a time. To produce each token, the GPU must read every weight of the model out of high-bandwidth memory, then do a small amount of arithmetic with it. For a single sequence, the amount of math per byte read is tiny. That ratio of math to bytes is called arithmetic intensity. When it is low, the chips spend most of their time waiting for memory, not computing. Decode at small batch has very low intensity, so it is memory-bandwidth bound. Because new GPUs add compute much faster than bandwidth, decode does not speed up nearly as much as the FLOP numbers suggest. The wall actually gets worse on newer hardware unless you change the algorithm. The fixes all reduce bytes moved or do more work per byte: quantize the weights to fewer bits, shrink the KV cache, and batch many requests so one weight read serves many tokens at once.

**The memory wall** is the divergence between two GPU growth curves. Peak compute roughly triples each generation. HBM bandwidth grows only about 1.6 times. So compute keeps outrunning the ability to feed it. What decides which one binds is arithmetic intensity, the FLOPs done per byte read. Decode at batch 1 reads all weights to emit one token. That puts intensity near one operation per byte, far below the GPU's break-even point. So decode is bandwidth-bound, not compute-bound. The numbers are stark. Intensity is: $$I = \frac{\text{FLOPs}}{\text{bytes moved}}$$ Reading a 70B model in fp16 moves about 140 GB. On an H100 at 3 TB/s that read takes roughly 47 ms. The matching arithmetic at 989 TFLOP/s finishes in under a millisecond. Memory is the binding cost by hundreds of times. The wall worsens on newer silicon. More FLOPs with only modestly more bandwidth pushes the break-even intensity higher. So low-intensity decode falls further behind. Every decode optimization attacks bytes moved, not FLOPs. Quantization cuts bytes per weight. GQA, MLA, and KV quantization shrink cache reads. Batching and speculative decoding amortize one weight read across more tokens.