For a 200-token prompt and 1000-token completion, why does decode dominate end-to-end latency?

Attempt it

A request has a 200-token prompt and asks for a 1000-token completion. Explain why decode dominates wall-clock latency even though prefill processes more characters per pass. Use a back-of-envelope comparison.

Free · 2 AI evals / day

Quick reference

TL;DR

Prefill is one parallel pass over the whole prompt, so it is fast. Decode is one sequential pass per output token, so 1000 tokens means 1000 trips and the wall-clock bulk.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine baking. Prefill is like mixing all your ingredients in one big bowl at once: you do it once and it is quick, no matter how many ingredients. Decode is like decorating a cake one sprinkle at a time, and each sprinkle has to wait for the previous one to land before you know where to place the next. With 1000 sprinkles, the decorating takes far longer than the mixing, even though mixing involved more stuff. The prompt is the mixing step. The completion is the decorating step. Every output word must wait for the word before it, so a long completion stacks up many tiny waits into most of the total time you feel.

Key concepts

Prefill versus decode Time to first token Time per output token Memory-bandwidth bound

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: prefill is one parallel pass setting TTFT, decode is N sequential passes setting TPOT, the data dependency, the HBM bandwidth gate, and a tokens per second back of envelope.

Real-world examples

Real products, models, and research that use this idea.

vLLM reports decode throughput and TTFT as separate metrics, reflecting the two-phase split that this question turns on.
Anthropic and OpenAI bill input and output tokens at different rates partly because output tokens carry the sequential decode cost.
NVIDIA TensorRT-LLM uses chunked prefill to interleave prompt processing with decode so long prompts do not stall the token stream.
Llama 4 and DeepSeek V4 serving stacks lean on GQA and MLA specifically to cut the per-decode-step HBM bytes that gate latency.
Speculative decoding in production serving verifies several draft tokens per target pass to attack decode's sequential bottleneck directly.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QAt what completion length does prefill latency stop being negligible relative to decode?

Sign in or go Pro to see the approach for this follow-up.

QHow does batching change which phase dominates and the per-token economics?

Sign in or go Pro to see the approach for this follow-up.

QWhy does speculative decoding help decode but not prefill?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Reasoning from character or token counts. Prefill touches more tokens but runs them in parallel in one pass; decode runs far fewer tokens but pays a full sequential pass for each one.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.