Each autoregressive decode step reads from HBM (constant per step) plus (which grows linearly with the live sequence length). This second term is why long-context decode gets slower as generation progresses.
Decode is autoregressive: the model produces one token, feeds it back in, produces the next token, and so on. Each of those steps has to bring two things from GPU memory (HBM) into the compute units.
The first thing is the model weights. These are the parameters the model learned during training. Their size is fixed by the model: a 70-billion-parameter model in FP16 is about 140 GB of weights. Every decode step has to load the same 140 GB to produce one token. It does not matter whether the model has generated 10 tokens so far or 10,000 tokens, the weights are the same every step.
The second thing is the KV cache. The KV cache stores the keys and values for every token already seen, both the prompt and the previously generated output. The new token's attention has to look at all those prior tokens, so the cache has to be on chip during the step. The catch is that the cache grows by one token's worth of K and V every step. At step 100 the cache holds 100 tokens worth. At step 10,000 it holds 10,000.
So per-token latency at decode time is roughly weight bytes plus KV cache bytes, divided by HBM bandwidth. At short context the weights dominate. At long context the KV cache grows large enough to dominate, and decode visibly slows down as generation progresses.
This is why decode is called memory-bandwidth-bound: the bottleneck is how fast you can move bytes from HBM into the compute units, not how fast you can do the math.
Each decode step in an autoregressive LLM moves two distinct sets of bytes from HBM to compute, and only the math on those bytes is on the critical path. Decode is fundamentally a memory-bandwidth-bound workload, and the two bytes-moved terms tell you everything about why.
The constant term is the full model weights. To produce one token, the model has to run through every layer once, and every layer's weight matrices must be in compute scope during the matmul. That is the entire parameter count of the model, regardless of context length or position in the generation. A 70B model in FP16 reads roughly 140 GB per decode step. In FP8 that drops to 70 GB. In FP4 (Blackwell-class) to 35 GB. The weight read is the same on token 1 of generation as on token 10,000 of generation: bytes-per-step is fixed by the model size and the dtype.
The growing term is the KV cache. Attention at position t needs to compute the score of the new query against every key for positions 1 through t-1, and then read the corresponding values. Those keys and values were computed during the prompt's prefill and during prior decode steps, and they are cached so the model does not recompute them. But they still must be moved from HBM into the attention kernel on every step, because each attention operation re-reads the full live K and V. The per-token bytes of the KV cache equals roughly 2 (K and V) times the number of layers times the number of KV heads times the head dimension times bytes per element. Multiply by the live sequence length and you get the total KV bytes the step has to load.
At short context (a few thousand tokens) the weight term dominates and the per-token latency is essentially constant: weights divided by HBM bandwidth. Around the crossover point (somewhere between 10k and 50k tokens depending on the model and the KV format) the KV cache becomes comparable to the weights, and per-token latency starts to climb visibly.
Quick reference
TL;DR
Each decode step reads the full model weights (constant) plus the live KV cache (grows linearly with context length). Decode is bandwidth-bound, and the KV term is what makes long-context decode slow.
Imagine the model is a chef cooking one dish at a time. Every dish, the chef has to grab the entire recipe book from a shelf, that is the model weights, and it is always the same weight no matter what dish. The chef also has to read every note they wrote down from earlier dishes in the meal, that is the KV cache, and after twenty dishes the stack of notes is twenty pages. After two hundred dishes it is two hundred pages. The recipe-book trip is constant per dish, but the notes-reading trip grows. By dish number two hundred, reading the notes takes longer than reading the recipe. That is why long stories get slower to write as they get longer.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Prefill rather than decode Prefill is matrix-matrix (one weight read amortized across many prompt tokens), so it is compute-bound and arithmetic intensity is high. The two-term decode analysis does not describe prefill cost.
Tiny models on huge GPUs where neither weight nor KV bytes are close to bandwidth-limited At very low utilization the workload may sit on kernel-launch overhead or other CPU-side bottlenecks rather than HBM bandwidth, and the two-term model is not the binding constraint.
Training rather than inference Training processes all positions in parallel and computes gradients; there is no autoregressive decode loop and the per-step bytes pattern looks like prefill.
Embedding or single-forward classification tasks There is no decode phase at all; the inference is one forward pass, behaves like prefill, and the two-term model does not apply.
Quick scan · 2–4 min
Everything important, quickly.
3 min: name both byte terms (weights constant, KV linear in T), quantify them on a typical 70B model, explain the crossover at long context, and name the optimization that targets each term.
Real products, models, and research that use this idea.
1Llama 3.1 70B with GQA-8 in FP8 loads roughly 70 GB of weights and about 32 GB of KV at 200k context per decode step on a single GPU, dominated by weight bandwidth at short context and by KV bandwidth at long context.
2DeepSeek V4 ships Multi-head Latent Attention specifically to compress the KV term, replacing per-head K and V with a low-rank latent that streams far fewer bytes per token at long context.
3vLLM's PagedAttention keeps the KV cache in fixed-size blocks so fragmentation does not waste bandwidth, but each decode step still reads all live blocks of all active sequences.
4FP8 KV cache (supported in vLLM and TensorRT-LLM) halves the KV bytes per token versus FP16, directly cutting long-context decode latency without changing the model.
5GPT-5.5 at 200k tokens shows visibly slower per-token decode than at 4k tokens, the dominant cause being the KV cache size growing far beyond the model weights term.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the KV cache load per step include both K and V even though the new token only adds one K and one V?
Sign in or go Pro to see the approach for this follow-up.
QHow does FP8 KV (versus FP16 KV) change the crossover point between weights and KV bytes?
Sign in or go Pro to see the approach for this follow-up.
QWhy does TP help the weight term but not the per-sequence KV term?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Saying decode only re-reads what it needs for that one new token. Wrong: each decode step reloads the entire weight matrix plus the entire current KV cache from HBM, because every output token must be computed against the full parameter set and the full attention history.