Compute the KV cache size for a 70B model at 128k context (80 layers, 64 heads, head_dim 128, FP16).

Attempt it

State the KV cache memory formula and compute it for a 70B-class model with 80 layers, 64 KV heads, head_dim 128, sequence length 128k tokens, FP16, batch size 1. Show units. How does the result compare to the weights themselves?

Free · 2 AI evals / day

Quick reference

TL;DR

KV cache bytes equal 2 times layers times KV heads times head_dim times sequence length times batch times dtype. At 128k context a 70B model's cache (~171 GB) exceeds its own weights.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a hotel that must keep a guest file for every person who has ever checked in, on every floor of the building. Each new guest adds one file per floor, and you can never throw any away while they stay. With a few guests the filing cabinet is tiny. With a hundred thousand guests across eighty floors, the cabinets fill the entire basement, far more space than the hotel's own furniture takes up. The KV cache is that filing cabinet for a language model. Every token it has read leaves a note on every layer, and the notes pile up linearly as the conversation grows. Eventually the notes need more memory than the model weights, which is why long chats get expensive to serve.

Key concepts

KV cache KV cache memory Grouped-query attention Long context serving

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: state the formula with all six factors plus the 2, compute ~171 GB, compare to ~140 GB weights, then name GQA, FP8 KV, and paged attention as the mitigations.

Real-world examples

Real products, models, and research that use this idea.

vLLM uses paged attention to allocate the KV cache in fixed blocks, recovering the 60 to 80 percent that contiguous allocation wastes at long context.
DeepSeek V4 ships Multi-head Latent Attention to compress K and V into a low-rank latent, shrinking the per-request cache by roughly an order of magnitude versus full MHA.
Llama 4 and Mistral Large 3 default to grouped-query attention with 8 KV heads, cutting the formula's KV head term 8x against the 64-head config in this question.
TensorRT-LLM on NVIDIA H100 and B200 ships FP8 KV cache plus chunked prefill as the production reference for long-context serving.
Anthropic's Claude Opus 4.7 serving uses cache-aware routing so requests sharing long system prompts colocate and reuse prefix KV state.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the cache scale linearly with batch size but the model weights do not?

Sign in or go Pro to see the approach for this follow-up.

QHow much does grouped-query attention with 8 KV heads change the 171 GB result?

Sign in or go Pro to see the approach for this follow-up.

QIf the cache outgrows the weights at 128k context, what sets the maximum batch size?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Dropping the per-layer factor or the factor of 2 for K and V. Both errors put the estimate off by one or two orders of magnitude, which hides that the cache outgrows the weights.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.