For a 200-token prompt and 1000-token completion, why does decode dominate end-to-end latency?

A generation request has two phases. **Prefill** reads the prompt; decode writes the completion. They feel different because of how the work is shaped, not how much there is. Prefill processes all 200 prompt tokens in a single forward pass. The GPU runs them in parallel, so prefill finishes in tens of milliseconds. This sets time to first token, often abbreviated TTFT. Decode is the opposite. The model generates one output token, feeds it back in, generates the next, and repeats. For a 1000-token completion that is 1000 separate forward passes, run strictly one after another. Why can't we parallelize decode? Because token number two depends on token number one. You cannot compute a word until you know the word before it. This data dependency forces the sequence. Each decode step is also slow for its own reason: it must read the entire weight matrix from memory just to produce one token. At a typical 50 tokens per second, 1000 tokens take about 20 seconds, versus 50 milliseconds for prefill. Decode wins by orders of magnitude.

**Decode dominates** because of pass shape, not token volume. Prefill is one parallel forward pass over all 200 prompt tokens. Decode is 1000 sequential passes, one per output token. Prefill runs every prompt token at once, so the GPU stays compute-bound and busy. It finishes in tens of milliseconds and sets time to first token. The 200 tokens cost almost nothing because they share a single pass. Decode cannot do that. Token i+1 depends on token i, so the passes must run in order. Each step reads the full weight matrix and the growing cache from HBM to emit just one token. That makes every step memory-bandwidth-bound, gated by a full memory round-trip. The back-of-envelope: at roughly 50 tokens per second, 1000 output tokens take about 20 seconds. Prefill was about 50 milliseconds. Decode is therefore around 400 times larger here, three orders of magnitude. The rule generalizes. Prefill scales with prompt length but runs in parallel. Decode scales with completion length and runs serially. Long outputs make decode the bulk of wall-clock time.