Prefill and decode: name the two phases of LLM inference and say which one is compute bound

Flashcard·Easy·4.0 · 0·~30s·Asked atCharacter Ai Deepseek Pinterest·Relevant atNVIDIA

Attempt it

The two phases are prefill and decode. Prefill processes all prompt tokens in parallel through every layer, producing the initial KV cache; this is a big matmul over a long sequence and saturates the tensor cores, so it is compute-bound. Decode then generates output tokens one at a time, each step reads the full KV cache, performs attention against it, and emits a single new token. Per-step work is tiny relative to cache size, so HBM bandwidth dominates and decode is memory bandwidth bound. That asymmetry is why TTFT (time to first token) and TPOT (time per output token) tune separately.

Your answer

**LLM inference has two distinct phases** with very different hardware profiles. Understanding the split is the first step in reasoning about serving performance. **Prefill.** When you submit a prompt, the model processes all prompt tokens in parallel through every layer. For a 1000-token prompt, the attention layer computes a 1000 x 1000 score matrix per head per layer, and the FFN sees 1000 token-vectors at once. The output of prefill is the initial KV cache (the K and V tensors for every prompt token at every layer) plus the first generated token. Prefill is **compute-bound**. The matrix multiplications saturate the GPU's tensor cores, and the bottleneck is FLOPS per second. On an H100, prefill can hit 500-700 TFLOPS sustained. **Decode.** After prefill, the model generates output tokens one at a time. Each decode step takes a single new token (the one just generated), computes Q, K, V for it (just one token's worth), and attends against the entire existing KV cache. The per-step compute is tiny: one query attending to all keys, instead of all queries attending to all keys. Decode is **memory bandwidth bound**. The bottleneck is not the matmul cost; it is the time to read the full KV cache from HBM into the SM for each decode step. On an H100, decode might use only 5-15% of the tensor cores but saturate the 3 TB/s HBM bandwidth. This asymmetry is why production LLM serving has two separate metrics: - **TTFT** (time to first token): dominated by prefill. - **TPOT** (time per output token): dominated by decode. And two separate sets of optimizations: kernel fusion and large batch sizes for prefill; KV-cache compression, quantization, and continuous batching for decode.

**Prefill and decode are the two phases of LLM inference,** and they live on opposite ends of the roofline model. **Prefill** processes all N prompt tokens in parallel through every transformer layer. The attention matmuls are O(N^2 * d) compute and produce the initial KV cache. With N = 1000 and a 70B model, the matmuls saturate the GPU's tensor cores: compute-bound. The output is the first generated token plus the KV cache. **Decode** generates output tokens one at a time. Each step takes 1 token in, computes Q, K, V for it, appends to the KV cache, attends against the entire cache, and emits one new token. The KV cache must be read from HBM every step: bandwidth-bound. The asymmetry is sharp. Prefill on an H100 might run at 500+ TFLOPS using ~80% of tensor cores. Decode might run at 50 TFLOPS using ~10% of tensor cores but saturating the 3 TB/s HBM bandwidth. **Why this matters for production.** - **TTFT** (time to first token) is dominated by prefill. Long prompts mean slow TTFT. - **TPOT** (time per output token) is dominated by decode. Total latency grows linearly with output length. - **Decode batches well** because each step is bandwidth-bound and adding requests amortizes the KV-cache read. Prefill batches poorly because each prompt is already compute-saturating. - **Continuous batching** (vLLM, TensorRT-LLM) handles the asymmetry at the scheduler level. KV-cache compression (FP8, INT4), GQA, PagedAttention, and speculative decoding all target decode bandwidth or the KV cache.

TL;DR

Prefill processes all prompt tokens in parallel and is compute-bound (saturates tensor cores). Decode generates one token at a time and is memory bandwidth bound (HBM read of the full KV cache per step).

Memory aid

Easy to grasp

Imagine ordering a meal at a restaurant. The kitchen has to do two very different jobs. First, when you place your whole order at once (the prompt), the chef can fire up every burner in parallel and cook all the dishes simultaneously; the bottleneck is how much heat the kitchen can put out. Then, once the meal is going, the server has to walk to the kitchen, pick up one dish, and bring it to your table, then walk back for the next; the bottleneck is no longer how fast the kitchen cooks but how fast the server can run. LLM inference works the same way: prefill is the parallel cooking burst, decode is the per-dish trip back and forth. The two have completely different bottlenecks, which is why they get tuned separately.

Key concepts

Prefill phase Decode phase KV cache

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The prefill vs decode split is the single most important mental model for reasoning about LLM serving performance. Almost every production optimization in modern LLM serving stacks (vLLM, TensorRT-LLM, SGLang, custom inference engines) is targeting either the prefill side or the decode side of this asymmetry, and confusing the two leads to optimizations that miss their intended target.

This deep dive walks the roofline picture that motivates the split, explains why prefill is compute-bound and decode is bandwidth-bound (and why this asymmetry is structural rather than implementational), maps the two phases to the production metrics TTFT and TPOT, and surveys the modern optimizations that target each side.

Mental model: prefill is a big parallel matmul that saturates tensor cores. Decode is a per-token loop that saturates HBM bandwidth. Different bottlenecks, different fixes.

The roofline picture

The roofline model is the standard tool for reasoning about whether a kernel is compute-bound or memory-bound. The two axes are FLOPS per second (peak compute) and arithmetic intensity (FLOPs per byte loaded from memory). The 'roof' is the minimum of peak compute and (peak bandwidth * arithmetic intensity).

Kernels with low arithmetic intensity are bandwidth-bound: the bottleneck is reading data, not computing. Kernels with high arithmetic intensity are compute-bound: the bottleneck is the FLOPS, not the bandwidth.

Prefill: high arithmetic intensity

Prefill processes N prompt tokens in parallel through every layer. The attention matmul is Q @ K^T of shape (N, d_head) @ (d_head, N), which is O(N^2 * d_head) compute. The bytes loaded are O(N * d_head) for Q, K, V. Arithmetic intensity is roughly N * d_head, which for N = 1000, d_head = 128 is 128k FLOPs per byte; deep in the compute-bound region of the roofline.

On an H100 (peak 989 TFLOPS FP16, peak 3 TB/s HBM), kernels need arithmetic intensity above 989/3 = 330 FLOPs/byte to be compute-bound. Prefill at N = 1000 has intensity ~128k FLOPs/byte. Decisively compute-bound.

Decode: low arithmetic intensity

Decode processes 1 token at a time. The attention compute is q @ K^T of shape (1, d_head) @ (d_head, N+T), which is O((N+T) * d_head) compute. The bytes loaded are O((N+T) * d_head) for the KV cache. Arithmetic intensity is roughly d_head, which for d_head = 128 is 128 FLOPs per byte; below the 330 FLOPs/byte threshold. Decisively bandwidth-bound.

Why this is structural

The ratio of compute to bytes loaded for attention scales like:

Prefill: N_query_tokens / 1 = N
Decode: 1 / 1 = 1

The ratio is N times larger for prefill than for decode, which is the entire reason for the asymmetry. As N grows, prefill becomes more compute-bound and decode becomes more bandwidth-bound. The structural fact is not a tuning issue; it cannot be optimized away.

Mapping to production metrics

Optimizations that target each side

How the asymmetry shows up in batch composition and scheduling

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

vLLM and TensorRT-LLM both separate prefill and decode at the scheduler level and use continuous batching to amortize decode bandwidth across requests.
Anthropic, OpenAI, and Google's production serving stacks publish TTFT and TPOT separately because they reflect different bottlenecks.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does decode batching scale throughput so dramatically while prefill batching does not?

Decode is bandwidth-bound: each request loads its KV cache from HBM independently, but the model weights are loaded once per step regardless of batch size. Adding requests amortizes the weight loading and improves arithmetic intensity. Prefill is already compute-bound at batch size 1 for moderate prompt lengths; adding more requests does not unlock additional compute, only adds queueing.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating LLM inference as one uniform workload and missing that prefill and decode have completely different bottlenecks (compute vs bandwidth) requiring different optimizations.

60 second bullets to scan on the way to the call.

Define the prefill phase in autoregressive LLM serving
Define the decode phase and how it differs from prefill

Primary sources. Browse if you want the original framing.