State the KV cache memory formula and compute it for a 70B class model with 80 layers, 64 KV heads, head_dim 128, sequence length 128k tokens, FP16, batch size 1. Show units. How does the result compare to the weights themselves?
KV cache bytes equal 2 times layers times KV heads times head_dim times sequence length times batch times bytes per element. At 128k context the cache (about 344 GB) is more than twice the weights.
Picture a hotel that must keep a guest file for every person who has ever checked in, on every floor of the building. Each new guest adds one file per floor, and you can never throw any away while they stay. With a few guests the filing cabinet is tiny. With a hundred thousand guests across eighty floors, the cabinets fill the entire basement, far more space than the hotel's own furniture takes up. The KV cache is that filing cabinet for a language model. Every token it has read leaves a note on every layer, and the notes pile up linearly as the conversation grows. Eventually the notes need more memory than the model weights, which is why long chats get expensive to serve.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The arithmetic here answers a capacity-planning point: at long context the KV cache, not the model weights, is the resource that runs out first. Get the formula right and the conclusion follows.
The KV cache stores the key and value vectors for every token the model has already processed, so that each new decode step can attend over history without recomputing past projections. Its footprint is the product of every dimension it spans, and the skill is listing those dimensions correctly without dropping any.
The two factors most often forgotten are the per-layer multiplier and the leading 2 for K and V. Dropping the first makes the answer 80 times too small; dropping the second halves it.
Deriving the formula from the cache structure
Build the formula by asking what the cache actually contains, rather than trying to recall a memorized number. For each transformer layer, for each KV head, for each token seen so far, the model stores one key vector and one value vector, each of length equal to the per-head dimension. Nothing else from the past needs to persist.
That enumeration is the whole derivation. The byte count is the product of all of those dimensions, times the bytes per element, times two for the pair of K and V tensors:
Every factor has a plain meaning. L is the layer count, because each transformer block keeps its own independent cache. The KV head count may be smaller than the query head count under grouped-query attention, which is how GQA shrinks the cache. The per-head dimension and the sequence length size the individual tensors. Bytes per element is 2 for FP16 or BF16, and 1 for FP8 or int8. Batch multiplies because the cache is per-request state, not shared like the weights.
Some things are not in the formula. Query vectors are not cached, because a past token's query is never re-read once its own attention step completes. Attention scores and outputs are not cached either, because they are recomputed every step from the new query against the stored keys. Counting any of these as cached state inflates the estimate.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM uses paged attention to allocate the KV cache in fixed 16-token blocks; its launch post reports 60 to 80 percent memory waste in earlier systems against under 4 percent with paging.
- DeepSeek-V2 uses Multi-head Latent Attention to compress K and V into a low-rank latent, and reports a 93.3 percent smaller KV cache than DeepSeek 67B.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the cache scale linearly with batch size but the model weights do not?
Weights are shared across all requests in a batch, so they are paid once. The cache is per-request state, so it multiplies by batch. This asymmetry is why memory-bound serving caps batch size, and why batching helps throughput only until the cache fills HBM.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Dropping the per-layer factor or the factor of 2 for K and V. The first makes the estimate 80 times too small and the second halves it; either hides how far the cache outgrows the weights.
60 second bullets to scan on the way to the call.
The full formula and the meaning of every factor including the leading 2
Why the cache scales linearly with both context length and batch size
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.