The two phases are prefill and decode. Prefill processes all prompt tokens in parallel through every layer, producing the initial KV cache; this is a big matmul over a long sequence and saturates the tensor cores, so it is compute-bound. Decode then generates output tokens one at a time, each step reads the full KV cache, performs attention against it, and emits a single new token. Per-step work is tiny relative to cache size, so HBM bandwidth dominates and decode is memory bandwidth bound. That asymmetry is why TTFT (time to first token) and TPOT (time per output token) tune separately.
**LLM inference has two distinct phases** with very different hardware profiles. Understanding the split is the first step in reasoning about serving performance.
**Prefill.** When you submit a prompt, the model processes all prompt tokens in parallel through every layer. For a 1000-token prompt, the attention layer computes a 1000 x 1000 score matrix per head per layer, and the FFN sees 1000 token-vectors at once. The output of prefill is the initial KV cache (the K and V tensors for every prompt token at every layer) plus the first generated token.
Prefill is **compute-bound**. The matrix multiplications saturate the GPU's tensor cores, and the bottleneck is FLOPS per second. On an H100, prefill can hit 500-700 TFLOPS sustained.
**Decode.** After prefill, the model generates output tokens one at a time. Each decode step takes a single new token (the one just generated), computes Q, K, V for it (just one token's worth), and attends against the entire existing KV cache. The per-step compute is tiny: one query attending to all keys, instead of all queries attending to all keys.
Decode is **memory bandwidth bound**. The bottleneck is not the matmul cost; it is the time to read the full KV cache from HBM into the SM for each decode step. On an H100, decode might use only 5-15% of the tensor cores but saturate the 3 TB/s HBM bandwidth.
This asymmetry is why production LLM serving has two separate metrics:
- **TTFT** (time to first token): dominated by prefill.
- **TPOT** (time per output token): dominated by decode.
And two separate sets of optimizations: kernel fusion and large batch sizes for prefill; KV-cache compression, quantization, and continuous batching for decode.
**Prefill and decode are the two phases of LLM inference,** and they live on opposite ends of the roofline model.
**Prefill** processes all N prompt tokens in parallel through every transformer layer. The attention matmuls are O(N^2 * d) compute and produce the initial KV cache. With N = 1000 and a 70B model, the matmuls saturate the GPU's tensor cores: compute-bound. The output is the first generated token plus the KV cache.
**Decode** generates output tokens one at a time. Each step takes 1 token in, computes Q, K, V for it, appends to the KV cache, attends against the entire cache, and emits one new token. The KV cache must be read from HBM every step: bandwidth-bound.
The asymmetry is sharp. Prefill on an H100 might run at 500+ TFLOPS using ~80% of tensor cores. Decode might run at 50 TFLOPS using ~10% of tensor cores but saturating the 3 TB/s HBM bandwidth.
**Why this matters for production.**
- **TTFT** (time to first token) is dominated by prefill. Long prompts mean slow TTFT.
- **TPOT** (time per output token) is dominated by decode. Total latency grows linearly with output length.
- **Decode batches well** because each step is bandwidth-bound and adding requests amortizes the KV-cache read. Prefill batches poorly because each prompt is already compute-saturating.
- **Continuous batching** (vLLM, TensorRT-LLM) handles the asymmetry at the scheduler level.
KV-cache compression (FP8, INT4), GQA, PagedAttention, and speculative decoding all target decode bandwidth or the KV cache.
Quick reference
TL;DR
Prefill processes all prompt tokens in parallel and is compute-bound (saturates tensor cores). Decode generates one token at a time and is memory bandwidth bound (HBM read of the full KV cache per step).
Imagine ordering a meal at a restaurant. The kitchen has to do two very different jobs. First, when you place your whole order at once (the prompt), the chef can fire up every burner in parallel and cook all the dishes simultaneously; the bottleneck is how much heat the kitchen can put out. Then, once the meal is going, the server has to walk to the kitchen, pick up one dish, and bring it to your table, then walk back for the next; the bottleneck is no longer how fast the kitchen cooks but how fast the server can run. LLM inference works the same way: prefill is the parallel cooking burst, decode is the per-dish trip back and forth. The two have completely different bottlenecks, which is why they get tuned separately.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Embedding-only or classification-only models that produce a single output and have no decode phase Encoder-only models like BERT have one forward pass and no autoregressive decode; the prefill vs decode asymmetry does not apply.
Training workloads where the model processes full sequences with teacher forcing Training is closer to prefill-style parallel processing throughout; the decode-style per-token loop is an inference-time concept.
Single-token continuations or completion APIs that generate only one or two tokens If generation is very short, decode latency is small and the asymmetry matters less; TTFT dominates total latency.
Workloads where prompts and outputs are both small (e.g., under 50 tokens each) Both phases are short and neither phase saturates its respective bottleneck; per-request overheads (kernel launch, scheduler) may dominate instead.
Quick scan · 2–4 min
Everything important, quickly.
Walk the roofline picture, explain prefill as a compute-bound parallel matmul over all prompt tokens, contrast decode as a bandwidth-bound per-token loop against the full KV cache, derive the TTFT-vs-TPOT split, and close with the modern optimizations (continuous batching, KV compression, speculative decoding, chunked prefill) that target each side of the asymmetry.
Real-world examples
Real products, models, and research that use this idea.
1vLLM and TensorRT-LLM both separate prefill and decode at the scheduler level and use continuous batching to amortize decode bandwidth across requests.
2Anthropic, OpenAI, and Google's production serving stacks publish TTFT and TPOT separately because they reflect different bottlenecks.
3Speculative decoding (used in Llama 4 Maverick and Gemini 3.1 Pro production stacks) batches multiple draft-model tokens into one verifier-model forward pass to recover compute utilization during decode.
4DeepSeek V4's MLA (multi-latent attention) compresses the KV cache via low-rank latents, directly attacking the decode-bandwidth bottleneck.
5Chunked prefill (vLLM 0.5+) interleaves long prefill with decode steps from other requests to smooth tail latency.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does decode batching scale throughput so dramatically while prefill batching does not?
Sign in or go Pro to see the approach for this follow-up.
QHow does speculative decoding recover compute utilization during decode?
Sign in or go Pro to see the approach for this follow-up.
QWhy does chunked prefill help tail latency?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating LLM inference as one uniform workload and missing that prefill and decode have completely different bottlenecks (compute vs bandwidth) requiring different optimizations.