Each decode step loads ___ from HBM, regardless of how many tokens have already been generated

Fill in blank·Easy·4.0 · 0·~1 min·Asked atMeesho Robust Intelligence Tata Digital·Relevant atNVIDIA

Attempt it

Each autoregressive decode step reads from HBM (constant per step) plus (which grows linearly with the live sequence length). This second term is why long context decode gets slower as generation progresses.

Decode is autoregressive: the model produces one token, feeds it back in, produces the next token, and so on. Each of those steps has to bring two things from GPU memory (HBM) into the compute units. The first thing is the model weights. These are the parameters the model learned during training. Their size is fixed by the model: a 70-billion-parameter model in FP16 is about 140 GB of weights. Every decode step has to load the same 140 GB to produce one token. It does not matter whether the model has generated 10 tokens so far or 10,000 tokens, the weights are the same every step. The second thing is the KV cache. The KV cache stores the keys and values for every token already seen, both the prompt and the previously generated output. The new token's attention has to look at all those prior tokens, so the cache has to be on chip during the step. The catch is that the cache grows by one token's worth of K and V every step. At step 100 the cache holds 100 tokens worth. At step 10,000 it holds 10,000. So per-token latency at decode time is roughly weight bytes plus KV cache bytes, divided by HBM bandwidth. At short context the weights dominate. At long context the KV cache grows large enough to dominate, and decode visibly slows down as generation progresses. This is why decode is called memory-bandwidth-bound: the bottleneck is how fast you can move bytes from HBM into the compute units, not how fast you can do the math.

Each decode step in an autoregressive LLM moves two distinct sets of bytes from HBM to compute, and only the math on those bytes is on the critical path. Decode is fundamentally a memory-bandwidth-bound workload, and the two bytes-moved terms tell you everything about why. The constant term is the full model weights. To produce one token, the model has to run through every layer once, and every layer's weight matrices must be in compute scope during the matmul. That is the entire parameter count of the model, regardless of context length or position in the generation. A 70B model in FP16 reads roughly 140 GB per decode step. In FP8 that drops to 70 GB. In FP4 (Blackwell-class) to 35 GB. The weight read is the same on token 1 of generation as on token 10,000 of generation: bytes-per-step is fixed by the model size and the dtype. The growing term is the KV cache. Attention at position t needs to compute the score of the new query against every key for positions 1 through t-1, and then read the corresponding values. Those keys and values were computed during the prompt's prefill and during prior decode steps, and they are cached so the model does not recompute them. But they still must be moved from HBM into the attention kernel on every step, because each attention operation re-reads the full live K and V. The per-token bytes of the KV cache equals roughly 2 (K and V) times the number of layers times the number of KV heads times the head dimension times bytes per element. Multiply by the live sequence length and you get the total KV bytes the step has to load. At short context (a few thousand tokens) the weight term dominates and the per-token latency is essentially constant: weights divided by HBM bandwidth. Around the crossover point (somewhere between 10k and 50k tokens depending on the model and the KV format) the KV cache becomes comparable to the weights, and per-token latency starts to climb visibly.

TL;DR

Each decode step reads the full model weights (constant) plus the live KV cache (grows linearly with context length). Decode is bandwidth-bound, and the KV term is what makes long-context decode slow.

Memory aid

Easy to grasp

Imagine the model is a chef cooking one dish at a time. Every dish, the chef has to grab the entire recipe book from a shelf, that is the model weights, and it is always the same weight no matter what dish. The chef also has to read every note they wrote down from earlier dishes in the meal, that is the KV cache, and after twenty dishes the stack of notes is twenty pages. After two hundred dishes it is two hundred pages. The recipe-book trip is constant per dish, but the notes-reading trip grows. By dish number two hundred, reading the notes takes longer than reading the recipe. That is why long stories get slower to write as they get longer.

Key concepts

Decode step KV cache HBM bandwidth Memory-bound workload Model weights

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Decode is two bytes-moved streams per step, not one. Internalizing both is the foundation for understanding every long-context optimization and every batching decision in production serving. The first is the model weights, fixed in size by the architecture and dtype, loaded in full on every step. The second is the KV cache, growing linearly with the live sequence length, also loaded in full on every step. The kernel is memory-bound, so per-token latency is essentially the sum of those bytes divided by HBM bandwidth.

This deep dive quantifies both terms on a realistic 70B model, derives the crossover where the KV term overtakes the weight term, explains why batching helps one term and not the other, and connects the math to the long-context optimization stack.

Why decode loads the full weights on every step

An autoregressive decoder produces one token at a time. To produce token t, the model runs a full forward pass with t-1 tokens of cached attention state and one new query at position t. Every layer's matmul has to operate on the current activation against the layer's weight matrix, and the weight matrix has to be in compute scope during that matmul.

The weight matrix lives in HBM. To bring it into compute (the tensor cores in shared memory and registers), the kernel issues memory operations that read the entire weight tensor. That happens on every layer of every step. There is no caching of weights across steps that avoids the HBM read; HBM is the cache, in this hierarchy.

The total weight bytes per step is roughly the parameter count times the bytes per parameter. For Llama 3.1 70B in FP16 that is 140 GB per step per GPU. In FP8 it is 70 GB. In FP4 (Blackwell-class) it is 35 GB. Tensor parallelism shards the weights across N GPUs, so the per-GPU read drops to weight_bytes / N. None of this changes step to step. Whether the model has produced 10 tokens or 10,000, the weight read is the same.

Why the KV cache grows and how big it actually is

Per-token latency on bandwidth-bound hardware

Why batching helps the weight term but not the KV term

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Term	Per-step bytes	Scaling	Optimizations
Model weights	Param count * bytes-per-param	Constant per step	Lower-precision dtype (FP8/FP4), tensor parallelism
KV cache	2 * L * H_kv * d_h * b * T	Linear in T (and in batch)	GQA / MQA / MLA, FP8/FP4 KV, sliding-window, paged attention

Real products, models, and research that use this idea.

Llama 3.1 70B with GQA-8 in FP8 loads roughly 70 GB of weights and about 32 GB of KV at 200k context per decode step on a single GPU, dominated by weight bandwidth at short context and by KV bandwidth at long context.
DeepSeek V4 ships Multi-head Latent Attention specifically to compress the KV term, replacing per-head K and V with a low-rank latent that streams far fewer bytes per token at long context.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the KV cache load per step include both K and V even though the new token only adds one K and one V?

Attention at step t computes softmax(Q_t * K_1..t-1^T) * V_1..t-1, so the kernel must touch every prior K and every prior V to compute the attention output. The new K and V are added to the cache, but the full cache is read every step.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Saying decode only re-reads what it needs for that one new token. Wrong: each decode step reloads the entire weight matrix plus the entire current KV cache from HBM, because every output token must be computed against the full parameter set and the full attention history.

60 second bullets to scan on the way to the call.

Name the two HBM byte streams per decode step.
Describe the structure of the weight term.

Primary sources. Browse if you want the original framing.