End-to-end latency: fill in the queue, prefill and decode components.

Fill in blank·Easy·4.0 · 0·~1 min

Attempt it

End-to-end latency for a single LLM completion producing N_out output tokens decomposes as: queue_time + + (N_out - 1) * . The first non-queue term is dominated by compute and is what users perceive as TTFT. The second term grows linearly with output length and is dominated by memory bandwidth. For long completions the second term typically the first.

**End-to-end latency** for an LLM request is the wall-clock time from when the client sends the request to when the last token arrives back. It decomposes cleanly into three pieces. First, **queue time**: how long the request waits on the server before a GPU slot is available. Under light load this is near zero; under heavy load it dominates p99 latency. Second, **TTFT** (time to first token): the time to process the entire prompt in one parallel prefill forward pass and emit the first output token. TTFT is paid once per request and scales with input length, not output length. Prefill is compute-bound on modern H100 and B200 GPUs. Third, **(N_out - 1) * TPOT** (time per output token), where N_out is the number of generated tokens. Each decode step produces one new token by reading the full weights and KV cache from HBM. Decode is bandwidth-bound. For long completions, this term dominates the total. The practical consequence: a 200-token answer with TTFT = 300ms and TPOT = 20ms takes about 0.3 + 199 * 0.02 ≈ 4.3 seconds. The decode portion (4 seconds) dwarfs the prefill portion (0.3 seconds). Modern serving stacks like vLLM and SGLang separate p50/p95 metrics for TTFT and TPOT for exactly this reason.

**End-to-end latency** = `queue_time + TTFT + (N_out - 1) * TPOT`. Each term has a different cost driver. **Queue time** is admission policy dependent. It is near zero on idle servers but can dominate p99 latency under load. Production stacks tune `max_num_batched_tokens`, prefill-decode worker ratios, and chunked prefill specifically to keep queue depth bounded. **TTFT** is dominated by prefill compute, which processes the entire input prompt in one parallel forward pass. Prefill is compute-bound on H100 / B200: arithmetic intensity is high because weights are read once and reused across all input tokens. TTFT scales with input length, not output length. Prefix caching (Anthropic 0.1x reads, OpenAI 0.5x reads on 1024+ token prefixes) collapses this to a fast KV-cache load. **TPOT** is dominated by HBM bandwidth. Each decode step rereads the full weights plus the accumulated KV cache, but operates on a single token's activations. Arithmetic intensity is low, the kernel is bandwidth-bound, and TPOT scales weakly with sequence length (via KV-cache size) and strongly with model size. For any reasonably long completion (say N_out >= 50), the decode term dominates total time. A 500-token answer with TTFT = 500ms and TPOT = 25ms takes 0.5 + 499 * 0.025 ≈ 13 seconds, where 12.5 seconds (96 percent) is decode. This is why streaming UIs feel snappy on TTFT but slow on overall response: the first token arrives fast, then the user watches the rest stream.

Quick reference

TL;DR

End-to-end latency decomposes into queue_time + TTFT + (N_out - 1) * TPOT, where TTFT is compute-bound prefill and TPOT is bandwidth-bound decode.

Memory aid

Easy to grasp

Picture sending a package through a busy mailroom. First it sits in line behind other packages: that is queue time. Then a clerk has to read the whole address (the entire prompt) before anything can move: that is TTFT, paid once. After that, the clerk hand-writes one stamp at a time onto the box, again and again, until it is fully labeled. Each stamp is the same small wait, but you pay it once per stamp. If the package needs hundreds of stamps, that second pile of waits ends up much bigger than the one-time address-reading. End-to-end latency is the sum of all three pieces.

Key concepts

Latency decomposition Time to first token Time per output token Prefill Decode

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: three-term decomposition + prefill compute-bound vs decode bandwidth-bound + decode dominates for N_out > 50 + production levers for each term + why TTFT and TPOT need separate SLOs.

Real-world examples

Real products, models, and research that use this idea.

vLLM v1's benchmark scripts report p50/p95/p99 for both TTFT and inter-token latency (effective TPOT) separately, plus queue depth.
Anthropic Claude Sonnet 4.6 dashboards alert on TTFT and total completion time as separate SLOs; long-context chat targets p95 TTFT under one second.
GPT-5.5 streaming endpoints surface TTFT via the first SSE delta timestamp; product teams chart it independent of total tokens emitted.
Llama 4 Maverick deployments on H100 fleets report TPOT around 25-40ms per token and TTFT around 500ms-1s at 4K context, depending on batch size.
DeepSeek V4 with speculative decoding shows TPOT compressed by 1.8-2.2x in published vLLM benchmarks while TTFT is unchanged.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf TTFT = 200ms and TPOT = 30ms, what is the breakeven output length where decode equals prefill?

QHow does prefill-decode disaggregation in vLLM v1 change the latency decomposition?

QWhy does TPOT slowly increase across a long completion rather than staying constant?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Lumping TTFT and per-token decode into one latency number. They move on different axes and need separate SLOs because prefill is compute-bound while decode is bandwidth-bound.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.