Across which axes is the KV cache stored, per layer, per head, per batch?

Attempt it

All three. Each transformer block has its own attention layer, so the KV cache holds a separate K and V tensor for every layer. Within a layer, each KV head (which under MHA equals the query-head count, under GQA is smaller, under MQA equals 1) stores its own slice. And every request in the batch keeps its own cache because tokens differ. Conceptual shape per layer is roughly (batch, num_kv_heads, seq_len, head_dim). Two K/V tensors per layer, multiplied by layer count, multiplied by batch size, gives total cache size, the dominant memory term at long context.

Your answer

Quick reference

TL;DR

The KV cache spans all three axes: per layer, per KV head, and per batch request. Each combination stores its own K and V tensor over the sequence.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a multistory library where every floor (a transformer layer) has its own card catalog. On each floor, several librarians (the heads) keep separate drawers of cards. And every visitor (a request in the batch) brings their own set of bookmarks. To find a book, you need the right floor, the right librarian's drawer, and your personal bookmark stack. The KV cache works the same way: layers, heads, and requests are all separate axes, and any one of them being missed will make your memory math wrong by an order of magnitude.

Key concepts

KV cache Multi-head attention

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3m: the three independent axes (layer, KV head, batch), how MHA vs GQA vs MQA changes the head axis only, the per-token formula, and how prefix sharing partially compresses the batch axis.

Real-world examples

Real products, models, and research that use this idea.

HuggingFace transformers' `past_key_values` is a tuple of length L, each entry holding K and V of shape (batch, H_kv, T, d_h).
vLLM's PagedAttention treats this cache as paged memory with per-request page tables and prefix sharing.
Llama 4 Maverick and Claude Opus 4.7 both use GQA, so num_kv_heads is much smaller than num_heads in their config files.
DeepSeek V4's MLA compresses K and V into a shared low rank latent that still spans the layer and batch axes.
llama.cpp stores the cache as a single contiguous buffer indexed by (layer, head, position, dim) and supports per-axis quantization.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does PagedAttention's prefix sharing change the batch-axis story?

Sign in or go Pro to see the approach for this follow-up.

QWhy is cross-layer KV sharing not yet a production default?

Sign in or go Pro to see the approach for this follow-up.

QWhat changes for MLA, where K and V are compressed into a latent?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Pretending the cache is per layer only, then sizing memory as if heads or batch share storage. Each axis is genuinely independent and contributes a multiplicative factor.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.