Why does prefill saturate compute while decode is bottlenecked on memory bandwidth?
Explain why the prefill phase of LLM inference saturates GPU compute, while the decode phase is bottlenecked by HBM memory bandwidth. Reference arithmetic intensity in your answer.
Prefill reuses one big weight read across many prompt tokens, so it saturates compute. Decode reads the same weights plus KV cache per single token, so it is limited by HBM bandwidth.
Imagine a chef with a huge cookbook. Prefill is like cooking dinner for 200 guests at once. The chef walks to the cookbook one time and reads the recipe, and that single trip pays off across hundreds of plates. Decode is like cooking one plate, then walking all the way back to reread the entire cookbook, then cooking one more plate, over and over. The chef spends almost all the time walking to and from the shelf, not cooking. The stove, which is the compute, sits mostly idle while the chef fetches pages. That walk to the bookshelf is reading from GPU memory, and it dominates decode.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Prefill and decode are the same model running two very different workloads. They sit on opposite sides of the roofline, so the first step before any optimization is to tell which regime a request is in. Cutting the wrong cost improves a number that is not the bottleneck.
Both regimes follow from arithmetic intensity: the number of useful floating point operations the GPU performs per byte it reads from high-bandwidth memory. GPUs can do far more arithmetic per second than they can move bytes per second. So any workload with low intensity is limited by memory long before it runs out of compute. The roofline model draws this as two lines: a sloped bandwidth ceiling on the left and a flat compute ceiling on the right. Intensity places a kernel on one side or the other.
Two serving techniques follow from the two regimes: chunked prefill smooths long prompts on a shared GPU, and prefill-decode disaggregation tunes each phase on its own hardware.
Arithmetic intensity and the roofline
Arithmetic intensity is FLOPs divided by HBM bytes moved. A GPU like an H100 or B200 has a fixed peak compute rate and a fixed peak memory bandwidth. Divide peak FLOPs by peak bandwidth and you get the machine balance point, often a few hundred FLOP per byte on these GPUs. It is the intensity at which a kernel moves from being limited by bytes to being limited by math.
The roofline model captures this as a formula for achievable throughput:
If your kernel's intensity is above the balance point, the first term wins and you are compute bound: the math units are the limit and you are using the memory system efficiently. If your intensity is below it, the second term wins and you are bandwidth bound: the GPU finishes the arithmetic and then sits idle waiting for the next bytes to arrive.
Plotted on a log-log chart, the left region is a diagonal set by bandwidth. The right region is a flat ceiling set by peak compute. A kernel's intensity drops it onto one region or the other. The gap between where it lands and the ceiling above it is wasted hardware. Prefill and decode are two kernels from the same model that land in these two regions, which is why one optimization rarely helps both equally.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM ships chunked prefill so a long prompt does not stall other requests' decode steps, smoothing inter-token latency under mixed load.
- DistServe and Mooncake run prefill and decode on separate GPU pools, each tuned for its own roofline regime, and transfer the KV cache between them.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does increasing the decode batch size move the workload along the roofline?
Batching reuses one weight read across many concurrent sequences, so FLOPs rise while the weight bytes stay fixed. That lifts arithmetic intensity and pushes decode toward the compute roof, until the KV cache bytes per step grow large enough to cap the achievable batch.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the whole request as one regime. Prefill and decode sit on opposite sides of the roofline, so a single FLOP cutting optimization helps one phase and does nothing for the other.
60 second bullets to scan on the way to the call.
Definition of arithmetic intensity and how it maps onto the roofline regimes
Why prefill amortizes one weight read across many tokens
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.