Decode is the autoregressive loop where one forward pass produces one token, appending its K and V to the cache and attending over the full cache to sample the next token.
Picture writing a long sentence one word at a time, where each new word has to consider every word you have already written. To save time you keep a small notebook of notes about every previous word, so you do not have to reread the whole story each step. You take the next blank space, jot down notes for the single new word you are about to add, glance over every note in the book, then write down what comes next. Then you repeat. The model does the same thing. Reading the prompt is one big batch operation; producing the answer is hundreds of these one-step writes. That is why long answers feel slow even when the prompt was short.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The decode phase is where the model actually produces text. Prefill reads the prompt in one big parallel matmul, but decode is the part the user perceives as the model thinking. Every word of every reply you have ever seen from an LLM came out of this loop, one token at a time.
Understanding decode is the foundation of understanding LLM serving. The reason serving costs scale with output length, the reason latency feels long even on fast hardware, the reason batching matters so much, the reason long-context models eat HBM, all of these trace back to mechanics of the decode loop.
This deep dive walks through what a decode step actually computes, why it is structured around a KV cache, why Q is treated asymmetrically from K and V, why the loop is sequential within one request, and why despite being slow per token decode is the right design choice given the autoregressive structure of language.
The goal by the end is that you can stand at a whiteboard, draw a decode step, name what reads from HBM, name what writes to HBM, name what gets reused next step, and connect it cleanly to the arithmetic intensity story and to why batching dominates serving throughput.
What happens in a single decode step
A decode step is one forward pass through the transformer that produces exactly one new token. The input to the step is the embedding of the most recently generated token plus the existing KV cache holding all previous tokens' keys and values.
Inside each transformer block, the model projects the new token's hidden state into three vectors: query Q_t, key K_t, and value V_t. The new K_t and V_t are written to the cache at position t. The new Q_t scores against the entire cached set of keys, producing attention weights over all previous positions. Those weights re-weight the cached values into a single output vector, which then flows through the MLP and on to the next layer.
At the top of the stack, the final hidden state is projected into vocabulary logits. A sampler (greedy argmax, top-p, top-k, temperature) picks one token. That token's embedding becomes the input for step t+1, and the loop repeats.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM continuous batching merges new requests into the in-flight decode batch every step on H100 and B200 clusters serving Llama 4 and DeepSeek V4.
- Anthropic streams Claude Opus 4.7 output token by token as the decode loop produces each token, exposing per-step timing in the API.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is Q computed fresh each decode step instead of cached like K and V?
Walk through the attention computation: Q_t is the query for the new token and is only used in that single step to score against the cached K. After that step it is never reused. K_t and V_t, by contrast, will be attended to by every future token, so they are worth keeping. Caching Q would store data that has no future reader.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating decode as a single matmul like prefill. Decode is N sequential forward passes, each producing one token, which is why output length dominates wall-clock time.
60 second bullets to scan on the way to the call.
The definition of decode as one forward pass per output token
Which projections are computed fresh per step and which are cached
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.