Describe what happens during the decode phase of LLM inference.

After the model reads the prompt in one parallel pass (the prefill), it enters the **decode phase**. This is where the answer is actually produced, one token at a time. Each decode step does a single forward pass through the model and emits exactly one new token. To produce that token, the model computes the query, key, and value vectors for just the latest position. The new K and V are appended to a structure called the KV cache, which already holds the K and V for every earlier token. The model then runs attention against the entire cache, producing logits for the next token. A sampler (greedy, top-p, temperature) picks one token from the distribution. That token becomes the input for the next decode step. A 1000-token reply runs this loop 1000 times sequentially. There is no way to parallelise it: each token depends on the previous one. This is why output length dominates the total time a user waits, and why decode optimisation is mostly about lowering per-token time and batching across users.

**Decode** is the autoregressive token by token generation loop that runs after prefill. Each step is one forward pass that produces exactly one new token. For the newly generated token at position t, the model computes Q, K, and V from its hidden state. The new K_t and V_t are appended to the KV cache, which already holds the projections for positions 0 to t-1. Then attention runs: Q_t against every cached K, producing a score over all previous tokens, softmaxed and used to weight the cached V values. The result feeds the rest of the layer stack, and the final logits go to the sampler. Q is computed once and discarded each step. K and V are kept forever, growing the cache by one token per step. This asymmetry is why we cache K and V but not Q. The loop is strictly sequential. Token at step t depends on the token sampled at step t-1, so you cannot decode tokens in parallel for one request. A 1000-token reply runs 1000 forward passes back to back. Decode is also bandwidth-bound at low batch. The forward pass mostly reads weights from HBM and does little arithmetic per byte, the canonical memory-bound kernel. This is why batching across users is the single biggest throughput lever in serving.