End to end latency: fill in the queue, prefill and decode components.

Fill in blank·Easy·4.0 · 0·~1 min·Asked atGraphcore Ltimindtree Palantir·Relevant atOpenAI

Attempt it

End to end latency for a single LLM completion producing N_out output tokens decomposes as: queue_time + + (N_out - 1) * . The first non-queue term is dominated by compute and is what users perceive as TTFT. The second term grows linearly with output length and is dominated by memory bandwidth. For long completions the second term typically the first.

**End-to-end latency** for an LLM request is the wall-clock time from when the client sends the request to when the last token arrives back. It decomposes cleanly into three pieces. First, **queue time**: how long the request waits on the server before a GPU slot is available. Under light load this is near zero; under heavy load it dominates p99 latency. Second, **TTFT** (time to first token): the time to process the entire prompt in one parallel prefill forward pass and emit the first output token. TTFT is paid once per request and scales with input length, not output length. Prefill is compute-bound on modern H100 and B200 GPUs. Third, **(N_out - 1) * TPOT** (time per output token), where N_out is the number of generated tokens. Each decode step produces one new token by reading the full weights and KV cache from HBM. Decode is bandwidth-bound. For long completions, this term dominates the total. The practical consequence: a 200-token answer with TTFT = 300ms and TPOT = 20ms takes about 0.3 + 199 * 0.02 ≈ 4.3 seconds. The decode portion (4 seconds) dwarfs the prefill portion (0.3 seconds). Modern serving stacks like vLLM and SGLang separate p50/p95 metrics for TTFT and TPOT for exactly this reason.

**End-to-end latency** = `queue_time + TTFT + (N_out - 1) * TPOT`. Each term has a different cost driver. **Queue time** is admission policy dependent. It is near zero on idle servers but can dominate p99 latency under load. Production stacks tune `max_num_batched_tokens`, prefill-decode worker ratios, and chunked prefill specifically to keep queue depth bounded. **TTFT** is dominated by prefill compute, which processes the entire input prompt in one parallel forward pass. Prefill is compute-bound on H100 / B200: arithmetic intensity is high because weights are read once and reused across all input tokens. TTFT scales with input length, not output length. Prefix caching (Anthropic 0.1x reads, OpenAI 0.5x reads on 1024+ token prefixes) collapses this to a fast KV-cache load. **TPOT** is dominated by HBM bandwidth. Each decode step rereads the full weights plus the accumulated KV cache, but operates on a single token's activations. Arithmetic intensity is low, the kernel is bandwidth-bound, and TPOT scales weakly with sequence length (via KV-cache size) and strongly with model size. For any reasonably long completion (say N_out >= 50), the decode term dominates total time. A 500-token answer with TTFT = 500ms and TPOT = 25ms takes 0.5 + 499 * 0.025 ≈ 13 seconds, where 12.5 seconds (96 percent) is decode. This is why streaming UIs feel snappy on TTFT but slow on overall response: the first token arrives fast, then the user watches the rest stream.

TL;DR

End-to-end latency decomposes into queue_time + TTFT + (N_out - 1) * TPOT, where TTFT is compute-bound prefill and TPOT is bandwidth-bound decode.

Memory aid

Easy to grasp

Picture sending a package through a busy mailroom. First it sits in line behind other packages: that is queue time. Then a clerk has to read the whole address (the entire prompt) before anything can move: that is TTFT, paid once. After that, the clerk hand-writes one stamp at a time onto the box, again and again, until it is fully labeled. Each stamp is the same small wait, but you pay it once per stamp. If the package needs hundreds of stamps, that second pile of waits ends up much bigger than the one-time address-reading. End-to-end latency is the sum of all three pieces.

Key concepts

Latency decomposition Time to first token Time per output token Prefill Decode

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

End-to-end latency is the most-watched, most-misunderstood metric in LLM serving. Engineers new to the space often quote a single 'response time' number, then get surprised when their chat UI feels broken at p95 even though the average looks fine. The fix is to stop treating latency as one number and start treating it as a sum of three structurally different terms.

This deep dive walks through the canonical decomposition queue + TTFT + (N_out - 1) * TPOT, why each term has a different cost driver, what production levers move each one, and why separate SLOs for TTFT and total completion time are non-negotiable on any serious chat product.

The three-term decomposition in detail

A single LLM request producing N_out output tokens spends time in exactly three places.

Queue time: the interval between request arrival at the server and the start of compute. On an idle server this is near zero. Under load, it is the dominant component of p95 and p99 latency because requests pile up behind running batches.
TTFT: the wall-clock cost to read the entire input prompt, run one parallel prefill forward pass, and emit the first output token. TTFT scales with input length T_in but not with output length. It is what a streaming client observes as the gap before the first character appears.
(N_out - 1) * TPOT: the autoregressive decode loop. Each of the remaining output tokens requires its own forward pass, and each pass costs roughly TPOT (a few tens of milliseconds on modern hardware). This term scales linearly with N_out.

The breakeven math

For TTFT around 300ms and TPOT around 25ms, decode equals prefill at N_out around 13. Past that, decode dominates. Most real-world completions are 50-1000 tokens, putting them firmly in the decode-dominated regime.

Why prefill is compute-bound and decode is bandwidth-bound

Production levers for each term

Why SLOs split and how production stacks expose this

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

vLLM v1's benchmark scripts report p50/p95/p99 for both TTFT and inter-token latency (effective TPOT) separately, plus queue depth.
Anthropic Claude Sonnet 4.6 dashboards alert on TTFT and total completion time as separate SLOs; long-context chat targets p95 TTFT under one second.

What an interviewer would ask next. Try answering before peeking at the approach.

QIf TTFT = 200ms and TPOT = 30ms, what is the breakeven output length where decode equals prefill?

Solve (N_out - 1) * 30ms = 200ms, giving N_out about 8. Past 8 output tokens, decode dominates. Most real completions are 50-500 tokens, so decode is the main cost driver almost always.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Lumping TTFT and per-token decode into one latency number. They move on different axes and need separate SLOs because prefill is compute-bound while decode is bandwidth-bound.

60 second bullets to scan on the way to the call.

The three-term decomposition: queue, TTFT, (N_out - 1) * TPOT
Why prefill is compute-bound and decode is bandwidth-bound

Primary sources. Browse if you want the original framing.