Explain the 'memory wall' as it applies to LLM decode. Why does HBM bandwidth dominate compute? How is the gap evolving across GPU generations, and what optimization classes attack it directly?
The memory wall is the widening gap between compute growth and HBM-bandwidth growth; decode at low batch reads more bytes than it does math, so it is bandwidth-bound and worsens on newer GPUs.
Imagine a chef who can chop ingredients incredibly fast, but everything sits in a pantry down a long hallway. To cook one dish the chef sprints the hallway to fetch every ingredient, then chops it in seconds. The chopping is not the bottleneck, the hallway trips are. Every new kitchen model gives the chef faster hands but the same slow hallway. So cooking one dish at a time barely speeds up. The fix is to cook many dishes per hallway trip, or carry lighter ingredients. That hallway is memory bandwidth, the chopping is compute, and one dish at a time is decoding a single sequence. Reading the model from memory dominates, not the arithmetic.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The memory wall is the most load-bearing single fact in LLM serving, and the one candidates most often get backwards. The instinct is to treat a GPU as a compute engine and ask how many FLOPs a model needs. For autoregressive decode that instinct is wrong. The binding resource is not arithmetic. It is how fast you can stream bytes out of high-bandwidth memory into the compute units.
The term names a hardware trend. Across accelerator generations, peak compute has grown roughly threefold per step while HBM bandwidth has grown only about 1.6-fold. The two curves diverge, and the gap compounds. A workload whose performance is set by bandwidth therefore captures a shrinking fraction of each generation's headline FLOP number.
This deep dive builds the idea from the roofline model, derives arithmetic intensity, works the concrete H100 numbers for a 70B decode step, explains why the wall sharpens rather than softens on newer silicon, and finally maps every major decode optimization onto the single quantity it manipulates. By the end you should be able to predict, from a model size and a GPU spec sheet, whether a given decode workload is bandwidth-bound and roughly by how much.
The roofline model and arithmetic intensity
The roofline model plots achievable throughput on the vertical axis against arithmetic intensity on the horizontal axis. Arithmetic intensity is the ratio of useful FLOPs performed to bytes moved from memory:
At low intensity, throughput rises along a line whose slope is the memory bandwidth: you are memory-bound, and every extra byte per second of bandwidth buys more work. At high intensity the curve flattens at the peak compute ceiling: you are compute-bound, and only more FLOP/s helps.
The two regimes meet at the ridge point, the intensity at which bandwidth and compute are exactly balanced. Its value is the compute ceiling divided by the bandwidth. For an H100 that is roughly 989 TFLOP/s over 3 TB/s, an intensity of a few hundred FLOPs per byte. Any workload below that number is bandwidth-bound; anything above it is compute-bound.
The power of the model is that it reduces a complicated performance question to one number you can compute on a napkin. Take the FLOPs a kernel does, divide by the bytes it must move, and compare the result to the ridge. You instantly know which ceiling you are hitting and therefore which optimizations can possibly help. If you are far below the ridge, faster arithmetic is wasted money. If you are above it, more bandwidth is wasted money. The discipline of always asking 'where on the roofline does this sit?' is what separates engineers who tune the right thing from those who chase the headline FLOP number.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM uses continuous batching so a single weight read serves dozens of concurrent sequences, directly raising arithmetic intensity on decode.
- DeepSeek V4 ships Multi-head Latent Attention to compress KV cache reads, attacking the bandwidth side of the memory wall at long context.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does batching help a bandwidth-bound workload so disproportionately?
One weight read can serve every sequence in the batch. FLOPs scale with batch but the bytes moved for weights stay roughly fixed, so arithmetic intensity rises linearly until you hit the roofline ridge and become compute-bound.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating decode as compute-bound and reaching for faster matmuls or more FLOPs. At batch 1 the GPU sits idle waiting on memory; adding compute does nothing until you raise arithmetic intensity.
60 second bullets to scan on the way to the call.
Define the memory wall as compute growth outrunning bandwidth growth
State the per-generation gap in rough multipliers for FLOPs versus bandwidth
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.