Why does prefill saturate compute while decode is bottlenecked on memory bandwidth?
Explain why the prefill phase of LLM inference saturates GPU compute, while the decode phase is bottlenecked by HBM memory bandwidth. Reference arithmetic intensity in your answer.
Prefill multiplies one big weight read across many prompt tokens, so it saturates compute; decode reads the same weights plus KV cache per single token, so it starves on HBM bandwidth.
Imagine a chef with a huge cookbook. Prefill is like cooking dinner for 200 guests at once: the chef walks to the cookbook one time, reads the recipe, and that single trip pays off across hundreds of plates. Decode is like cooking one plate, then walking all the way back to reread the entire cookbook, then cooking one more plate, over and over. The chef spends almost all the time walking to and from the shelf, not actually cooking. The stove, which is the compute, sits mostly idle while the chef fetches pages. That walk to the bookshelf is reading from GPU memory, and decode is dominated by it.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: define arithmetic intensity and the roofline, contrast prefill (amortized weight read, compute bound, TTFT) with decode (per-token weight and KV re-read, bandwidth bound, TPOT), then the diagnostic recipe plus chunked prefill and disaggregation.
Real products, models, and research that use this idea.
- vLLM ships chunked prefill so a long prompt does not stall other requests' decode steps, smoothing inter-token latency under mixed load.
- DistServe and Mooncake disaggregate prefill and decode onto separate GPU pools, each tuned for its own roofline regime, shuttling the KV cache between them.
- NVIDIA Dynamo and TensorRT-LLM on B200 expose prefill-decode disaggregation plus chunked prefill as production knobs for serving Llama 4 and DeepSeek V4.
- DeepSeek V4's Multi-head Latent Attention shrinks the per-token KV bytes that decode must stream, directly attacking the bandwidth wall.
- SGLang batches and shares prefixes during prefill to lift arithmetic intensity, while continuous batching keeps decode amortized across requests.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does increasing the decode batch size move the workload along the roofline?
QWhy can a single very long prompt hurt the inter-token latency of unrelated requests sharing the GPU?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating the whole request as one regime. Prefill and decode sit on opposite sides of the roofline, so a single FLOP cutting optimization helps one phase and does nothing for the other.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.