Per-token decode time grows linearly as a chat session lengthens, debug it

Short answer·Medium·4.0 · 0·~3 min·Relevant atAnthropic NVIDIA

Attempt it

An interactive chat product reports that TPOT starts at ~40ms and climbs to ~120ms by the time a session reaches 30k tokens of history. Diagnose the cause from first principles, then list the four most impactful mitigations a serving team can ship without retraining the model. Quantify each lever's effect where you can.

Free · 2 AI evals / day

Decoding one token at a time is bandwidth-bound: each step reads the model's weights once from HBM, then reads each request's KV cache, and does a small amount of arithmetic. The wall-clock time per token (TPOT) is set by how many bytes you have to stream from memory. The weight term is fixed (say 140 GB for a 70B model in FP16). The KV term grows linearly with sequence length T. At 1k tokens of history the cache is tiny; at 30k tokens it has caught up to and can rival the weights. So total bytes per step roughly doubles or triples, and TPOT moves from 40 ms to 120 ms. Four mitigations a serving team can ship without retraining: 1. **Prefix-cache the system prompt and stable preamble.** A 30k context shared across turns can be re-used; the engine reads the cached KV blocks instead of recomputing them. 2. **Switch to a model with GQA or MLA.** GQA-8 shrinks KV bytes by 8x. MLA (DeepSeek V3/V4 style) compresses KV to a shared latent, 10 to 100x smaller. 3. **Sliding window with attention sinks** (StreamingLLM). Keep the first 4 to 8 tokens and the last W tokens, evict the middle. KV memory caps at a constant. 4. **Cap session length or summarise old history.** Hard ceiling on T means predictable TPOT. A fifth lever for cost-aware teams is FP8 or INT4 KV-cache quantisation, which halves or quarters cache bytes directly. The invariant: shrink bytes per step, shrink TPOT.

Decode is memory-bandwidth-bound. The bytes streamed from HBM each step decompose into weights (fixed) plus per-request KV cache (linear in T). On a 70B model the weight read is around 140 GB; the KV term at 30k tokens, with standard multi-head, is in the same ballpark. So bytes-per-step roughly doubles or triples between 1k and 30k context, and since TPOT is bandwidth-bound, it follows the same curve. The 40 to 120 ms reading is exactly what the formula predicts. Four mitigations a serving team can ship today. **1. Prefix cache the system prompt and stable preamble.** Long sessions share enormous amounts of context across turns. Engines like vLLM, SGLang, and TensorRT-LLM share these KV blocks across requests with reference counting, collapsing re-prefill cost to near zero and freeing HBM for larger batches. **2. Switch to a model with GQA or MLA.** GQA-8 (Llama family) shrinks the KV term roughly 8x. MLA (DeepSeek V3/V4) compresses KV to a shared low-rank latent, with effective shrinkage of 10 to 100x. This is the single biggest lever if model choice is on the table. **3. Sliding window with attention sinks (StreamingLLM).** Keep the first 4 to 8 tokens (the sinks that absorb attention mass) and the last W tokens. KV memory caps at O(W), TPOT stops growing entirely. The trade is middle-context recall. **4. Cap session length or compact older history.** Hard ceiling on T, predictable TPOT. Ships in a day. Loses fidelity but works for the long tail of sessions that rarely benefit from full history. Fifth lever for completeness: FP8 or INT4 KV cache. Halves or quarters cache bytes with under 0.5 percentage point quality cost on most benchmarks. The invariant: TPOT tracks bytes-moved-per-step. Every effective lever shrinks that quantity.

Quick reference

TL;DR

TPOT tracks bytes streamed per decode step.

Memory aid

Easy to grasp

Picture a librarian who has to fetch every book on a long shelf before answering each question, then re-shelve them. At first the shelf is short and each question is fast. As more questions come in the shelf grows, and the librarian's arms cover more distance every single time. The time to answer one question grows in lockstep with the shelf length. The fix is not to make the librarian faster, but to shorten the shelf. You can let them skip the books they have already memorised (prefix cache), use thinner books (compressed cache), remove the middle of the shelf and keep only the start and the recent end (sliding window), or just cap how long the shelf is allowed to grow. The shelf length is what sets the speed.

Key concepts

KV cache Time per output token Prefix caching Grouped-query attention Multi-head latent attention (MLA)

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

6 min: decompose bytes-per-step into weights and KV + show KV grows linearly with T + walk through prefix caching, GQA/MLA, StreamingLLM, history compaction + FP8 KV as a fifth lever + tie it all to the bytes-per-step invariant.

Real-world examples

Real products, models, and research that use this idea.

vLLM and SGLang ship automatic prefix caching across requests with content-hashed paged KV blocks as a default in 2026.
Llama 4 Maverick uses GQA-8, dropping KV memory roughly 8x versus full multi-head attention at the same model size.
DeepSeek V3 and V4 use MLA to compress KV to a shared 512-dim latent, giving 10 to 100x effective KV shrinkage and serving 128k context cheaply.
Anthropic Claude prompt caching and OpenAI cached input pricing are product surfaces of the same engine-level prefix-caching mechanism.
StreamingLLM (Xiao et al. 2023) introduced the attention-sink trick used in many long-context chat stacks to keep KV bounded.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does prefix caching shrink HBM pressure even more than naive cache reuse suggests?

QHow does MLA achieve 10 to 100x KV shrinkage without obvious quality loss?

QWhat is the safe window size W for StreamingLLM in practice?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Blaming the slowdown on compute or model size, when the model itself is unchanged. The variable is sequence length T, and the cost it changes is HBM bytes per step through the KV-cache term, not the weight term.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.