Why does prefill saturate compute while decode is bottlenecked on memory bandwidth?

Attempt it

Explain why the prefill phase of LLM inference saturates GPU compute, while the decode phase is bottlenecked by HBM memory bandwidth. Reference arithmetic intensity in your answer.

Free · 2 AI evals / day

Quick reference

TL;DR

Prefill multiplies one big weight read across many prompt tokens, so it saturates compute; decode reads the same weights plus KV cache per single token, so it starves on HBM bandwidth.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a chef with a huge cookbook. Prefill is like cooking dinner for 200 guests at once: the chef walks to the cookbook one time, reads the recipe, and that single trip pays off across hundreds of plates. Decode is like cooking one plate, then walking all the way back to reread the entire cookbook, then cooking one more plate, over and over. The chef spends almost all the time walking to and from the shelf, not actually cooking. The stove, which is the compute, sits mostly idle while the chef fetches pages. That walk to the bookshelf is reading from GPU memory, and decode is dominated by it.

Key concepts

Arithmetic intensity Roofline model Prefill vs decode Memory bandwidth

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: define arithmetic intensity and the roofline, contrast prefill (amortized weight read, compute bound, TTFT) with decode (per-token weight and KV re-read, bandwidth bound, TPOT), then the diagnostic recipe plus chunked prefill and disaggregation.

Real-world examples

Real products, models, and research that use this idea.

vLLM ships chunked prefill so a long prompt does not stall other requests' decode steps, smoothing inter-token latency under mixed load.
DistServe and Mooncake disaggregate prefill and decode onto separate GPU pools, each tuned for its own roofline regime, shuttling the KV cache between them.
NVIDIA Dynamo and TensorRT-LLM on B200 expose prefill-decode disaggregation plus chunked prefill as production knobs for serving Llama 4 and DeepSeek V4.
DeepSeek V4's Multi-head Latent Attention shrinks the per-token KV bytes that decode must stream, directly attacking the bandwidth wall.
SGLang batches and shares prefixes during prefill to lift arithmetic intensity, while continuous batching keeps decode amortized across requests.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does increasing the decode batch size move the workload along the roofline?

Sign in or go Pro to see the approach for this follow-up.

QWhy can a single very long prompt hurt the inter-token latency of unrelated requests sharing the GPU?

Sign in or go Pro to see the approach for this follow-up.

QWhat does prefill decode disaggregation cost, and when is the tradeoff worth it?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating the whole request as one regime. Prefill and decode sit on opposite sides of the roofline, so a single FLOP cutting optimization helps one phase and does nothing for the other.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.