In LLM serving, what is the primary driver of end to end latency for a generation request?

**LLM generation** produces one token at a time. To produce token T+1, the model has to compute attention over all previous T tokens. That work scales with T, and you do it once per output token. For a 250-token response on a 70B model, you do 250 sequential decode steps. Each step reads the entire **KV cache** from HBM (high-bandwidth GPU memory), runs a matmul against the new token's query, and writes a new K and V row to the cache. The bottleneck is not the matmul itself; it is the memory bandwidth used to stream the KV cache. This is why long contexts get slow. At 8k context the per token decode is one number; at 128k context the cache is 16x larger and per token decode is correspondingly slower. The other options describe steps that are real but tiny. Network round trip is 5 to 50 ms total, regardless of output length. Tokenization is microseconds. Embedding lookup is part of a single forward pass, dominated by the rest of the network. Only option B (output tokens times per token decode time, memory-bandwidth-bound on the KV cache) actually scales with the work being done.

**End-to-end latency** for an LLM generation has two parts. `latency = TTFT + N * inter_token_latency` TTFT is time to first token. It depends on the prefill pass over the prompt and any retrieved context. Inter-token latency is per-output-token decode time. That step is memory-bandwidth-bound on the KV cache. Each decode step does the following. Take the new token's hidden state and compute Q, K, V. Append K and V to the cache. Then perform attention by streaming the entire cache from HBM. The arithmetic intensity of this is low (just a few matmuls per byte of cache read), so it sits on the memory side of the roofline, not the compute side. For a Llama 4 70B model on H100 at 8k context, per token decode is roughly 20 to 40 ms. A 250-token response takes 5 to 10 seconds, all of it dominated by KV cache streaming. The distractors are real costs but tiny by comparison. Network is 5 to 50 ms total. Tokenization is sub-millisecond on the CPU. Embedding lookup is a small fraction of one prefill pass. This is why every LLM serving optimisation in 2026 targets KV cache bandwidth: GQA, MQA, MLA, paged attention, fp8 KV, speculative decoding.