State the KV cache memory formula and compute it for a 70B class model with 80 layers, 64 KV heads, head_dim 128, sequence length 128k tokens, FP16, batch size 1. Show units. How does the result compare to the weights themselves?
KV cache bytes equal 2 times layers times KV heads times head_dim times sequence length times batch times dtype. At 128k context a 70B model's cache (~171 GB) exceeds its own weights.
Picture a hotel that must keep a guest file for every person who has ever checked in, on every floor of the building. Each new guest adds one file per floor, and you can never throw any away while they stay. With a few guests the filing cabinet is tiny. With a hundred thousand guests across eighty floors, the cabinets fill the entire basement, far more space than the hotel's own furniture takes up. The KV cache is that filing cabinet for a language model. Every token it has read leaves a note on every layer, and the notes pile up linearly as the conversation grows. Eventually the notes need more memory than the model weights, which is why long chats get expensive to serve.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This question looks like arithmetic, but it is really a capacity-planning question in disguise. The interviewer wants to know whether you understand that at long context the KV cache, not the model weights, is the resource that runs out first. Get the formula right and the conclusion follows automatically.
The KV cache stores the key and value vectors for every token the model has already processed, so that each new decode step can attend over history without recomputing past projections. Its footprint is the product of every dimension it spans, and the whole skill here is enumerating those dimensions correctly and not dropping any of them.
The two factors candidates forget are the per-layer multiplier and the leading 2 for K and V. Each omission moves the answer by an order of magnitude or more. This walkthrough states the formula, computes it for the given 70B configuration, compares it to the weight footprint, and then explains the architectural and serving responses that exist precisely because the raw number is so large.
Deriving the formula from the cache structure
Build the formula by asking what the cache actually contains, rather than trying to recall a memorized number. For each transformer layer, for each KV head, for each token seen so far, the model stores one key vector and one value vector, each of length equal to the per-head dimension. Nothing else from the past needs to persist.
That enumeration is the whole derivation. The byte count is the product of all of those dimensions, times the bytes per element, times two for the pair of K and V tensors:
Every factor has a plain meaning. L is the layer count, because each transformer block keeps its own independent cache. The KV head count may be smaller than the query head count under grouped-query attention, which is exactly the lever the architecture exploits. The per-head dimension and the sequence length size the individual tensors. Bytes per element is 2 for FP16 or BF16, and 1 for FP8 or int8. Batch multiplies because the cache is per-request state, not shared like the weights.
It is worth being explicit about what is NOT in the formula. Query vectors are not cached, because a past token's query is never re-read once its own attention step completes. Attention scores and outputs are not cached either, because they are recomputed every step from the new query against the stored keys. Confusing any of these with cached state is the fastest way to inflate the estimate and reveal a shaky mental model.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- vLLM uses paged attention to allocate the KV cache in fixed blocks, recovering the 60 to 80 percent that contiguous allocation wastes at long context.
- DeepSeek V4 ships Multi-head Latent Attention to compress K and V into a low-rank latent, shrinking the per-request cache by roughly an order of magnitude versus full MHA.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the cache scale linearly with batch size but the model weights do not?
Weights are shared across all requests in a batch, so they are paid once. The cache is per-request state, so it multiplies by batch. This asymmetry is why memory-bound serving caps batch size, and why batching helps throughput only until the cache fills HBM.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Dropping the per-layer factor or the factor of 2 for K and V. Both errors put the estimate off by one or two orders of magnitude, which hides that the cache outgrows the weights.
60 second bullets to scan on the way to the call.
The full formula and the meaning of every factor including the leading 2
Why the cache scales linearly with both context length and batch size
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.