Same topic, related formats. Practice these next.
Explain the 'memory wall' as it applies to LLM decode. Why does HBM bandwidth dominate compute? How is the gap evolving across GPU generations, and what optimization classes attack it directly?
The memory wall is the widening gap between compute growth and HBM-bandwidth growth; decode at low batch reads more bytes than it does math, so it is bandwidth-bound and worsens on newer GPUs.
Imagine a chef who can chop ingredients incredibly fast, but everything sits in a pantry down a long hallway. To cook one dish the chef sprints the hallway to fetch every ingredient, then chops it in seconds. The chopping is not the bottleneck, the hallway trips are. Every new kitchen model gives the chef faster hands but the same slow hallway. So cooking one dish at a time barely speeds up. The fix is to cook many dishes per hallway trip, or carry lighter ingredients. That hallway is memory bandwidth, the chopping is compute, and one dish at a time is decoding a single sequence. Reading the model from memory dominates, not the arithmetic.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: define the wall as compute and bandwidth divergence, derive arithmetic intensity, work the H100 70B weight-read versus arithmetic numbers, then map the four optimization classes onto the roofline.
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating decode as compute-bound and reaching for faster matmuls or more FLOPs. At batch 1 the GPU sits idle waiting on memory; adding compute does nothing until you raise arithmetic intensity.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.