**LLM generation** produces one token at a time. To produce token T+1, the model has to compute attention over all previous T tokens. That work scales with T, and you do it once per output token.
For a 250-token response on a 70B model, you do 250 sequential decode steps. Each step reads the entire **KV cache** from HBM (high-bandwidth GPU memory), runs a matmul against the new token's query, and writes a new K and V row to the cache. The bottleneck is not the matmul itself; it is the memory bandwidth used to stream the KV cache.
This is why long contexts get slow. At 8k context the per token decode is one number; at 128k context the cache is 16x larger and per token decode is correspondingly slower.
The other options describe steps that are real but tiny. Network round trip is 5 to 50 ms total, regardless of output length. Tokenization is microseconds. Embedding lookup is part of a single forward pass, dominated by the rest of the network. Only option B (output tokens times per token decode time, memory-bandwidth-bound on the KV cache) actually scales with the work being done.
**End-to-end latency** for an LLM generation has two parts.
`latency = TTFT + N * inter_token_latency`
TTFT is time to first token. It depends on the prefill pass over the prompt and any retrieved context. Inter-token latency is per-output-token decode time. That step is memory-bandwidth-bound on the KV cache.
Each decode step does the following. Take the new token's hidden state and compute Q, K, V. Append K and V to the cache. Then perform attention by streaming the entire cache from HBM. The arithmetic intensity of this is low (just a few matmuls per byte of cache read), so it sits on the memory side of the roofline, not the compute side.
For a Llama 4 70B model on H100 at 8k context, per token decode is roughly 20 to 40 ms. A 250-token response takes 5 to 10 seconds, all of it dominated by KV cache streaming.
The distractors are real costs but tiny by comparison. Network is 5 to 50 ms total. Tokenization is sub-millisecond on the CPU. Embedding lookup is a small fraction of one prefill pass.
This is why every LLM serving optimisation in 2026 targets KV cache bandwidth: GQA, MQA, MLA, paged attention, fp8 KV, speculative decoding.
Quick reference
TL;DR
Output length times per token decode time, where each step is bottlenecked by streaming the entire KV cache from HBM. Network, tokenization, and embedding lookups are negligible.
Imagine the model writing a long letter, one word at a time. Before each word, it has to read back every word it has already written from a giant wall of sticky notes. The reading speed is set by how fast you can pull notes off the wall, not by how fast you can think of the next word. Long letters are slow because the wall keeps growing; the cost of writing each new word is mostly the reading. The other things people worry about (sending the request, splitting it into pieces) take milliseconds and do not move the clock much.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very short outputs (1 to 5 tokens) where TTFT dominates For tiny generations, the prefill time over the input prompt is the whole latency budget. Optimise prefill, not decode.
Long-context prefill workloads (summarisation of 100k-token documents) When input is huge and output is small, the per token decode argument flips. Prefill compute dominates; FlashAttention and chunked prefill matter more.
Edge inference on small models (under 3B params) On small models, the KV cache is tiny and compute per layer becomes comparable; latency is no longer purely memory-bound.
Batch-API workloads where total throughput dominates per request latency Batch APIs trade latency for cost. Per-request latency is irrelevant; tokens-per-second per GPU is the metric.
Quick scan · 2–4 min
Everything important, quickly.
4 min: TTFT vs inter-token + memory-bandwidth-bound decode math + GQA, MQA, MLA cache reductions + fp8 KV + speculative decoding + 2026 production stacks.
Real-world examples
Real products, models, and research that use this idea.
1vLLM (UC Berkeley) hits 15 to 30 ms per token on Llama 4 70B with GQA and paged attention on H100, the production reference.
2SGLang ships continuous batching plus prefix caching for high throughput serving with sub second TTFT on long prompts.
3TensorRT-LLM on NVIDIA B200 takes per token decode below 10 ms for 70B-class models using fp8 KV cache and speculative decoding.
4Anthropic's Claude serving stack achieves first-token latency under 500 ms for typical chat workloads via aggressive cache aware routing.
5Together.ai exposes per token decode latency as a first class SLO in their serving dashboard for fine-tuned Llama 4 deployments.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does prefill latency differ from decode latency?
Sign in or go Pro to see the approach for this follow-up.
QWhat's the practical effect of GQA on per token latency?
Sign in or go Pro to see the approach for this follow-up.
QWhy does speculative decoding help so much in 2026 serving?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Blaming network round trip or tokenization for LLM latency. Those add a few milliseconds. The dominant cost is per token decode, which is bound by memory bandwidth on the KV cache.