What does the KV cache store, and why does it avoid Q?

Attempt it

Quick reference

TL;DR

The KV cache stores K and V for every past token at every layer and head; Q isn't cached because each token's Q is only used at its own step and never re-read later.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a long group conversation where you keep one sticky note per person summarizing what they said and what they care about. When a new person speaks, they glance at every sticky note to decide what to say, but the fleeting thought that prompted their own question evaporates the moment they finish speaking. The sticky notes are like K and V (read again and again by future speakers); the fleeting thought is like Q (used once, then gone).

Key concepts

KV cache Autoregressive decoding

Quick scan · 2–4 min

Everything important, quickly.

5m: causal attention directionality, what is and isn't cached, the per-token memory formula, prefill vs decode bottlenecks, and the GQA/MQA/MLA + PagedAttention chain that follows from cache pressure.

Code example

python

# Conceptual decode step with a KV cache.
# past_k, past_v: [batch, n_kv_heads, seq_so_far, d_head]
# x_new: hidden state of the new token, shape [batch, 1, d_model]

def decode_step(x_new, past_k, past_v, W_q, W_k, W_v, W_o):
    q_new = x_new @ W_q   # [B, 1, n_heads * d_head]
    k_new = x_new @ W_k   # [B, 1, n_kv_heads * d_head]
    v_new = x_new @ W_v   # [B, 1, n_kv_heads * d_head]

    # Append new K, V to the cache. No past Q is stored.
    k = concat([past_k, reshape(k_new)], dim=2)
    v = concat([past_v, reshape(v_new)], dim=2)

    # Attention: only the new Q against the full cached K, V.
    scores = (q_new @ k.transpose(-1, -2)) / sqrt(d_head)
    attn = softmax(scores, dim=-1)
    out = attn @ v
    return out @ W_o, k, v   # return updated cache

Real-world examples

Real products, models, and research that use this idea.

HuggingFace transformers exposes the cache via past_key_values; you pass it in and receive an updated cache out at each generation step.
vLLM's PagedAttention is a production KV-cache allocator using OS-style paging to eliminate fragmentation and unlock multi-tenant prefix sharing.
Llama 4 Maverick, Claude Opus 4.7, and Gemini 3.1 Pro all use grouped query attention to shrink the KV footprint relative to vanilla multi-head attention.
DeepSeek V4 uses Multi-head Latent Attention, compressing the cache into a low rank latent space and decompressing per step for further memory savings.
llama.cpp keeps the cache in a contiguous buffer and supports KV quantization formats such as Q8_0 and Q4_0 for memory constrained deployments.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the cache differ between prefill and decode in a production serving stack?

Sign in or go Pro to see the approach for this follow-up.

QWhat does the cache look like across a batch of requests with very different lengths?

Sign in or go Pro to see the approach for this follow-up.

QWhy doesn't MLA simply replace GQA everywhere if it shrinks the cache further?

Sign in or go Pro to see the approach for this follow-up.

QHow do speculative decoding and prefix sharing complicate cache management?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.