Each decode step loads ___ from HBM, regardless of how many tokens have already been generated

Fill in blank·Easy·4.0 · 0·~1 min

Attempt it

Each autoregressive decode step reads from HBM (constant per step) plus (which grows linearly with the live sequence length). This second term is why long-context decode gets slower as generation progresses.

Decode is autoregressive: the model produces one token, feeds it back in, produces the next token, and so on. Each of those steps has to bring two things from GPU memory (HBM) into the compute units. The first thing is the model weights. These are the parameters the model learned during training. Their size is fixed by the model: a 70-billion-parameter model in FP16 is about 140 GB of weights. Every decode step has to load the same 140 GB to produce one token. It does not matter whether the model has generated 10 tokens so far or 10,000 tokens, the weights are the same every step. The second thing is the KV cache. The KV cache stores the keys and values for every token already seen, both the prompt and the previously generated output. The new token's attention has to look at all those prior tokens, so the cache has to be on chip during the step. The catch is that the cache grows by one token's worth of K and V every step. At step 100 the cache holds 100 tokens worth. At step 10,000 it holds 10,000. So per-token latency at decode time is roughly weight bytes plus KV cache bytes, divided by HBM bandwidth. At short context the weights dominate. At long context the KV cache grows large enough to dominate, and decode visibly slows down as generation progresses. This is why decode is called memory-bandwidth-bound: the bottleneck is how fast you can move bytes from HBM into the compute units, not how fast you can do the math.

Each decode step in an autoregressive LLM moves two distinct sets of bytes from HBM to compute, and only the math on those bytes is on the critical path. Decode is fundamentally a memory-bandwidth-bound workload, and the two bytes-moved terms tell you everything about why. The constant term is the full model weights. To produce one token, the model has to run through every layer once, and every layer's weight matrices must be in compute scope during the matmul. That is the entire parameter count of the model, regardless of context length or position in the generation. A 70B model in FP16 reads roughly 140 GB per decode step. In FP8 that drops to 70 GB. In FP4 (Blackwell-class) to 35 GB. The weight read is the same on token 1 of generation as on token 10,000 of generation: bytes-per-step is fixed by the model size and the dtype. The growing term is the KV cache. Attention at position t needs to compute the score of the new query against every key for positions 1 through t-1, and then read the corresponding values. Those keys and values were computed during the prompt's prefill and during prior decode steps, and they are cached so the model does not recompute them. But they still must be moved from HBM into the attention kernel on every step, because each attention operation re-reads the full live K and V. The per-token bytes of the KV cache equals roughly 2 (K and V) times the number of layers times the number of KV heads times the head dimension times bytes per element. Multiply by the live sequence length and you get the total KV bytes the step has to load. At short context (a few thousand tokens) the weight term dominates and the per-token latency is essentially constant: weights divided by HBM bandwidth. Around the crossover point (somewhere between 10k and 50k tokens depending on the model and the KV format) the KV cache becomes comparable to the weights, and per-token latency starts to climb visibly.

Quick reference

TL;DR

Each decode step reads the full model weights (constant) plus the live KV cache (grows linearly with context length). Decode is bandwidth-bound, and the KV term is what makes long-context decode slow.

Memory aid

Easy to grasp

Imagine the model is a chef cooking one dish at a time. Every dish, the chef has to grab the entire recipe book from a shelf, that is the model weights, and it is always the same weight no matter what dish. The chef also has to read every note they wrote down from earlier dishes in the meal, that is the KV cache, and after twenty dishes the stack of notes is twenty pages. After two hundred dishes it is two hundred pages. The recipe-book trip is constant per dish, but the notes-reading trip grows. By dish number two hundred, reading the notes takes longer than reading the recipe. That is why long stories get slower to write as they get longer.

Key concepts

Decode step KV cache HBM bandwidth Memory-bound workload Model weights

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: name both byte terms (weights constant, KV linear in T), quantify them on a typical 70B model, explain the crossover at long context, and name the optimization that targets each term.

Side-by-side

Term	Per-step bytes	Scaling	Optimizations
Model weights	Param count * bytes-per-param	Constant per step	Lower-precision dtype (FP8/FP4), tensor parallelism
KV cache	2 * L * H_kv * d_h * b * T	Linear in T (and in batch)	GQA / MQA / MLA, FP8/FP4 KV, sliding-window, paged attention

Real-world examples

Real products, models, and research that use this idea.

Llama 3.1 70B with GQA-8 in FP8 loads roughly 70 GB of weights and about 32 GB of KV at 200k context per decode step on a single GPU, dominated by weight bandwidth at short context and by KV bandwidth at long context.
DeepSeek V4 ships Multi-head Latent Attention specifically to compress the KV term, replacing per-head K and V with a low-rank latent that streams far fewer bytes per token at long context.
vLLM's PagedAttention keeps the KV cache in fixed-size blocks so fragmentation does not waste bandwidth, but each decode step still reads all live blocks of all active sequences.
FP8 KV cache (supported in vLLM and TensorRT-LLM) halves the KV bytes per token versus FP16, directly cutting long-context decode latency without changing the model.
GPT-5.5 at 200k tokens shows visibly slower per-token decode than at 4k tokens, the dominant cause being the KV cache size growing far beyond the model weights term.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the KV cache load per step include both K and V even though the new token only adds one K and one V?

QHow does FP8 KV (versus FP16 KV) change the crossover point between weights and KV bytes?

QWhy does TP help the weight term but not the per-sequence KV term?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying decode only re-reads what it needs for that one new token. Wrong: each decode step reloads the entire weight matrix plus the entire current KV cache from HBM, because every output token must be computed against the full parameter set and the full attention history.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.