Output length times per token decode time. Each decode step is memory-bandwidth-bound: it streams the weights and the KV cache from HBM. Network, tokenization, and embedding lookups are negligible.
Imagine the model writing a long letter, one word at a time. Before each word, it has to reread its whole rulebook (the model's weights) and every word it has already written, kept on a giant wall of sticky notes. The reading speed is set by how fast you can pull pages and notes, not by how fast you can think of the next word. The cost of each new word is mostly the reading. Long letters make the wall bigger, so each word gets slower. The other things people worry about (sending the request, splitting it into pieces) take milliseconds and do not move the clock much.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LLM serving latency is the most common interview topic in modern system design because it concentrates so much of the production reality of running models at scale. The single most important fact to internalise is that decoding is memory-bound, not compute-bound. Once that is clear, every optimisation in the modern serving stack (GQA, paged attention, fp8 KV, speculative decoding) falls into place as a direct response to the same underlying constraint.
This deep dive walks the decomposition of end to end latency, the memory-bandwidth math, the architectural and serving-side responses, and the typical 2026 production stack. By the end you should be able to do the napkin math on per token latency for any model size and explain why network round trip and tokenization are red herrings.
Decomposing end to end latency
End-to-end latency for a generation request breaks into three components:
latency = network_RTT + TTFT (time to first token) + N_output * inter_token_latency
Network round trip is typically 5 to 50 ms total, a one-time cost per request. TTFT is the time to process the input prompt plus any retrieved context plus the first decode step; it is roughly proportional to input length and dominated by the prefill forward pass.
Inter-token latency is the time per subsequent output token. For Llama 3 70B with fp8 weights on two H100s, this is roughly 15 to 30 ms per token at 8k context. For 250 output tokens, the inter-token segment contributes 4 to 8 seconds.
The ratio matters. For long-output generations (chat completions, code generation), the inter-token segment dominates by an order of magnitude. For short-output generations (classification, single-answer Q-and-A), TTFT matters more. The question specifies a 'generation request' implying meaningful output, so option B (output tokens times per token decode) is the right driver.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM (UC Berkeley) hits 15 to 30 ms per token on Llama 3 70B with GQA and paged attention on two H100s, the production reference.
- SGLang ships continuous batching plus prefix caching for high throughput serving with sub second TTFT on long prompts.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does prefill latency differ from decode latency?
Prefill processes the entire input prompt in one parallel pass, dominated by compute (lots of matmuls in parallel). Decode is autoregressive, one token at a time, dominated by memory bandwidth (streaming the weights and the cache). TTFT is mostly prefill plus the first decode step; inter-token latency is pure decode.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blaming network round trip or tokenization for LLM latency. Those add a few milliseconds. The dominant cost is per token decode, which is bound by memory bandwidth: reading the weights and the KV cache each step.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.