Output length times per token decode time, where each step is bottlenecked by streaming the entire KV cache from HBM. Network, tokenization, and embedding lookups are negligible.
Imagine the model writing a long letter, one word at a time. Before each word, it has to read back every word it has already written from a giant wall of sticky notes. The reading speed is set by how fast you can pull notes off the wall, not by how fast you can think of the next word. Long letters are slow because the wall keeps growing; the cost of writing each new word is mostly the reading. The other things people worry about (sending the request, splitting it into pieces) take milliseconds and do not move the clock much.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LLM serving latency is the most common interview topic in modern system design because it concentrates so much of the production reality of running models at scale. The single most important fact to internalise is that decoding is memory-bound, not compute-bound. Once that is clear, every optimisation in the modern serving stack (GQA, paged attention, fp8 KV, speculative decoding) falls into place as a direct response to the same underlying constraint.
This deep dive walks the decomposition of end to end latency, the memory-bandwidth math, the architectural and serving-side responses, and the typical 2026 production stack. By the end you should be able to do the napkin math on per token latency for any model size and explain why network round trip and tokenization are red herrings.
Decomposing end to end latency
End-to-end latency for a generation request breaks into three components:
latency = network_RTT + TTFT (time to first token) + N_output * inter_token_latency
Network round trip is typically 5 to 50 ms total, a one-time cost per request. TTFT is the time to process the input prompt plus any retrieved context plus the first decode step; it is roughly proportional to input length and dominated by the prefill forward pass.
Inter-token latency is the time per subsequent output token. For Llama 4 70B on H100, this is roughly 15 to 30 ms per token at 8k context. For 250 output tokens, the inter-token segment contributes 4 to 8 seconds.
The ratio matters. For long-output generations (chat completions, code generation), the inter-token segment dominates by an order of magnitude. For short-output generations (classification, single-answer Q-and-A), TTFT matters more. The question specifies a 'generation request' implying meaningful output, so option B (output tokens times per token decode) is the right driver.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM (UC Berkeley) hits 15 to 30 ms per token on Llama 4 70B with GQA and paged attention on H100, the production reference.
- SGLang ships continuous batching plus prefix caching for high throughput serving with sub second TTFT on long prompts.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does prefill latency differ from decode latency?
Prefill processes the entire input prompt in one parallel pass, dominated by compute (lots of matmuls in parallel). Decode is autoregressive, one token at a time, dominated by memory bandwidth (streaming the cache). TTFT is mostly prefill plus the first decode step; inter-token latency is pure decode.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blaming network round trip or tokenization for LLM latency. Those add a few milliseconds. The dominant cost is per token decode, which is bound by memory bandwidth on the KV cache.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.