Across which axes is the KV cache stored, per layer, per head, per batch?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

All three. Each transformer block has its own attention layer, so the KV cache holds a separate K and V tensor for every layer. Within a layer, each KV head (which under MHA equals the query-head count, under GQA is smaller, under MQA equals 1) stores its own slice. And every request in the batch keeps its own cache because tokens differ. Conceptual shape per layer is roughly (batch, num_kv_heads, seq_len, head_dim). Two K/V tensors per layer, multiplied by layer count, multiplied by batch size, gives total cache size, the dominant memory term at long context.

Your answer

Quick reference

TL;DR

The KV cache spans all three axes: per layer, per KV head, and per batch request. Each combination stores its own K and V tensor over the sequence.

Memory aid

Easy to grasp

Imagine a multistory library where every floor (a transformer layer) has its own card catalog. On each floor, several librarians (the heads) keep separate drawers of cards. And every visitor (a request in the batch) brings their own set of bookmarks. To find a book, you need the right floor, the right librarian's drawer, and your personal bookmark stack. The KV cache works the same way: layers, heads, and requests are all separate axes, and any one of them being missed will make your memory math wrong by an order of magnitude.

Key concepts

KV cache Multi-head attention

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3m: the three independent axes (layer, KV head, batch), how MHA vs GQA vs MQA changes the head axis only, the per-token formula, and how prefix sharing partially compresses the batch axis.

Real-world examples

Real products, models, and research that use this idea.

HuggingFace transformers' `past_key_values` is a tuple of length L, each entry holding K and V of shape (batch, H_kv, T, d_h).
vLLM's PagedAttention treats this cache as paged memory with per-request page tables and prefix sharing.
Llama 4 Maverick and Claude Opus 4.7 both use GQA, so num_kv_heads is much smaller than num_heads in their config files.
DeepSeek V4's MLA compresses K and V into a shared low rank latent that still spans the layer and batch axes.
llama.cpp stores the cache as a single contiguous buffer indexed by (layer, head, position, dim) and supports per-axis quantization.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does PagedAttention's prefix sharing change the batch-axis story?

QWhy is cross-layer KV sharing not yet a production default?

QWhat changes for MLA, where K and V are compressed into a latent?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Pretending the cache is per layer only, then sizing memory as if heads or batch share storage. Each axis is genuinely independent and contributes a multiplicative factor.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.