An interactive chat product reports that TPOT starts at ~40ms and climbs to ~120ms by the time a session reaches 30k tokens of history. Diagnose the cause from first principles, then list the four most impactful mitigations a serving team can ship without retraining the model. Quantify each lever's effect where you can.
TPOT tracks bytes streamed per decode step.
Picture a librarian who has to fetch every book on a long shelf before answering each question, then re-shelve them. At first the shelf is short and each question is fast. As more questions come in the shelf grows, and the librarian's arms cover more distance every single time. The time to answer one question grows in lockstep with the shelf length. The fix is not to make the librarian faster, but to shorten the shelf. You can let them skip the books they have already memorised (prefix cache), use thinner books (compressed cache), remove the middle of the shelf and keep only the start and the recent end (sliding window), or just cap how long the shelf is allowed to grow. The shelf length is what sets the speed.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Time per output token (TPOT) is one of the two latency metrics that defines an LLM serving SLO, alongside time to first token. The observation in this question, TPOT climbing from 40 ms to 120 ms as a chat session reaches 30k tokens of history, is one of the most common production complaints on long-context deployments. The shape of the complaint tells you the diagnosis before any profiling: a linear climb that tracks session length points unambiguously at the KV cache.
The productive way to debug this class of issue is not to dive into kernel profiles but to write down the bandwidth budget and ask which term in it scales with the variable that changed. Decode is famously bandwidth-bound. The bytes streamed from HBM per step decompose cleanly into a fixed weight term and a linear KV-cache term. Once you have that decomposition, every mitigation becomes a question of which term to shrink and by how much.
This deep dive walks through the decomposition, derives the actual numbers that explain a 3x TPOT climb on a 70B model at 30k context, ranks the four most impactful mitigations by realistic shipping effort, and surfaces the underlying invariant that organises the entire mitigation menu. The goal by the end is to be able to look at any TPOT regression report and immediately identify which term in the bandwidth budget is to blame.
The bandwidth budget and why TPOT scales with it
Each decode step on an autoregressive transformer reads two distinct things from HBM: the model weights, and the KV cache for every active request.
The weight read is fixed per step. For a 70B model in FP16 it is approximately 140 GB. In FP8 it halves to 70 GB. In FP4 it halves again to 35 GB. Whatever the precision, it does not depend on how long the conversation has been running.
The KV-cache read is per-request and grows linearly with the number of tokens already generated. The formula is:
where L is layer count, H_kv is the number of KV heads (= H for MHA, H/G for GQA-G), d_h is head dimension, T is current sequence length, and b is bytes per element. For Llama 3.1 70B with GQA-8 in FP16: 80 layers, 8 KV heads, 128 head dim, 2 bytes. That gives roughly 320 KB per token, or about 10 GB at T = 30k.
TPOT = bytes-per-step / HBM-bandwidth. At constant bandwidth, every byte you can remove from per-step memory pressure shrinks TPOT proportionally. This is the single most important invariant in serving optimisation, and it organises every mitigation below.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM and SGLang ship automatic prefix caching across requests with content-hashed paged KV blocks as a default in 2026.
- Llama 4 Maverick uses GQA-8, dropping KV memory roughly 8x versus full multi-head attention at the same model size.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does prefix caching shrink HBM pressure even more than naive cache reuse suggests?
In a paged engine, identical KV blocks across requests are shared by reference count, not duplicated. A 30k system prompt across 1000 active sessions occupies one set of blocks, not 1000 sets. The HBM that would have been consumed by per-session duplication is now free for larger batches, compounding the win.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blaming the slowdown on compute or model size, when the model itself is unchanged. The variable is sequence length T, and the cost it changes is HBM bytes per step through the KV-cache term, not the weight term.
60 second bullets to scan on the way to the call.
The bandwidth-bound decomposition of bytes per step into weights and KV cache
Why the KV-cache term scales linearly with sequence length T
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.