Which phase governs the wall clock time of a long LLM generation?
Decode dominates wall-clock time on long generations because it runs one sequential, memory bandwidth bound forward pass per output token.
Imagine reading a whole letter at a glance versus writing your reply one word at a time. Reading the letter is quick because your eyes take it in all at once. Writing the reply is slow because each word has to come after the last. An LLM works the same way. Reading the prompt (prefill) happens in one fast parallel sweep. Writing the answer (decode) happens token by token, and each token needs its own slow step — so a long answer is what eats most of the time.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
"Where does the time go in an LLM call?" sounds like trivia until you have to hit a latency SLA. Get the phase model wrong and you optimize the part that was never the bottleneck — shaving milliseconds off tokenization while users wait seconds on decode.
The core insight is that a single request is really two workloads stapled together, and they stress completely different parts of the GPU. One reads the prompt in a parallel burst; the other writes the answer one token at a time. They have different bottlenecks, different scaling, and different optimization levers.
This walkthrough separates the two phases by their hardware behavior, shows why decode scales with output length while prefill is paid once, maps each phase to the latency metrics you report, and then complicates the simple story with the prompt-heavy workloads where the bottleneck flips.
Two phases, opposite hardware behavior
Every generation request runs prefill first, then decode. Prefill takes the full prompt and runs it through the model in one shot, computing the hidden states and the KV cache for all prompt tokens at once. Because every prompt position is available up front, the GPU does big matrix-matrix multiplies and stays busy — it is compute-bound and uses the hardware well.
Decode is the autoregressive loop. The model emits one token, appends it to the sequence, and runs again to emit the next. Token t+1 genuinely cannot start before token t exists, so there is no parallelism across output positions. Each step is a matrix-vector multiply that streams the entire weight set out of HBM just to produce a single token, which makes decode memory bandwidth bound with low compute utilization.
That contrast is the whole question. Prefill is fast but paid once; decode is slow but paid per token. Recognizing that they are different workloads — not two halves of the same workload — is what the question is really probing.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | Prefill | Decode |
|---|---|---|
| Parallelism | All prompt tokens at once | One token per pass, sequential |
| Bottleneck | Compute-bound (FLOPs) | Memory bandwidth bound (weight + KV reads) |
| Cost scales with | Prompt length, paid once | Output length, paid per token |
| Dominates wall-clock when | Huge prompt, short answer | Long answer (the common case) |
Real products, models, and research that use this idea.
- vLLM uses continuous batching and PagedAttention specifically to raise decode throughput.
- OpenAI and Anthropic chat APIs stream tokens during decode to mask the per-token latency from users.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does prompt to output ratio decide whether prefill or decode dominates?
Compare prefill cost (one parallel pass over the prompt) against decode cost (per-token pass times output length).
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a long prompt is the latency bottleneck — prefill scans the prompt in one parallel pass, while the per-token decode loop is what accumulates.
60 second bullets to scan on the way to the call.
Definition of prefill and decode as the two inference phases
Why prefill is compute-bound and parallel
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.