Why does prefill saturate compute while decode is bottlenecked on memory bandwidth?
Explain why the prefill phase of LLM inference saturates GPU compute, while the decode phase is bottlenecked by HBM memory bandwidth. Reference arithmetic intensity in your answer.
Prefill multiplies one big weight read across many prompt tokens, so it saturates compute; decode reads the same weights plus KV cache per single token, so it starves on HBM bandwidth.
Imagine a chef with a huge cookbook. Prefill is like cooking dinner for 200 guests at once: the chef walks to the cookbook one time, reads the recipe, and that single trip pays off across hundreds of plates. Decode is like cooking one plate, then walking all the way back to reread the entire cookbook, then cooking one more plate, over and over. The chef spends almost all the time walking to and from the shelf, not actually cooking. The stove, which is the compute, sits mostly idle while the chef fetches pages. That walk to the bookshelf is reading from GPU memory, and decode is dominated by it.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Prefill and decode are the same model running two completely different workloads. They sit on opposite sides of the roofline, and the single most valuable instinct an inference engineer can develop is telling which regime a given request lives in before touching any optimization. Cut the wrong cost and you spend weeks improving a number nobody is bottlenecked on.
The organizing idea is arithmetic intensity: the number of useful floating point operations the GPU performs per byte it reads from high-bandwidth memory. Modern accelerators can do far more arithmetic per second than they can move bytes per second, so any workload with low intensity is starved by memory long before it runs out of compute. The roofline model draws this as two lines, a sloped bandwidth ceiling on the left and a flat compute ceiling on the right, with intensity placing your kernel on one side or the other.
This deep dive defines arithmetic intensity precisely, derives why prefill lands in the compute bound region and decode in the bandwidth bound region, gives you a one-line diagnostic to classify any workload, and then connects each regime to the serving techniques built to address it: chunked prefill for smoothing long prompts and prefill-decode disaggregation for tuning each phase on its own hardware.
Arithmetic intensity and the roofline
Arithmetic intensity is FLOPs divided by HBM bytes moved. A GPU like an H100 or B200 has a fixed peak compute rate and a fixed peak memory bandwidth. Divide peak FLOPs by peak bandwidth and you get the machine balance point, often a few hundred FLOP per byte on current hardware. That single number is the hinge of the whole analysis: it is the intensity at which a kernel transitions from being limited by bytes to being limited by math.
The roofline model captures this as a formula for achievable throughput:
If your kernel's intensity is above the balance point, the first term wins and you are compute bound: the math units are the limit and you are using the memory system efficiently. If your intensity is below it, the second term wins and you are bandwidth bound: the GPU finishes the arithmetic and then sits idle waiting for the next bytes to arrive.
Plotted on a log-log chart, the left region is a diagonal set by bandwidth; the right region is a flat ceiling set by peak compute. A kernel's intensity drops it onto one region or the other, and the gap between where it lands and the ceiling above it is wasted hardware. The entire prefill versus decode story is just two kernels from the same model landing in those two regions, which is why a single optimization can never be right for both.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM ships chunked prefill so a long prompt does not stall other requests' decode steps, smoothing inter-token latency under mixed load.
- DistServe and Mooncake disaggregate prefill and decode onto separate GPU pools, each tuned for its own roofline regime, shuttling the KV cache between them.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does increasing the decode batch size move the workload along the roofline?
Batching reuses one weight read across many concurrent sequences, so FLOPs rise while the weight bytes stay fixed. That lifts arithmetic intensity and pushes decode toward the compute roof, until the KV cache bytes per step grow large enough to cap the achievable batch.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the whole request as one regime. Prefill and decode sit on opposite sides of the roofline, so a single FLOP cutting optimization helps one phase and does nothing for the other.
60 second bullets to scan on the way to the call.
Definition of arithmetic intensity and how it maps onto the roofline regimes
Why prefill amortizes one weight read across many tokens
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.