Compute the KV cache size for a 70B model at 128k context (80 layers, 64 heads, head_dim 128, FP16).

During autoregressive decoding, a transformer must attend to every previous token. To avoid recomputing each past token's key and value at every step, it stores them in the **KV cache**. The cache holds one K and one V vector per token, per layer, per KV head. That structure tells you the size. Multiply together every dimension the cache spans: the number of layers, the number of KV heads, the per-head dimension, the sequence length, and the batch size. Then multiply by the bytes per element, and by 2 because you store both K and V. For the given model: 2 times 80 layers times 64 heads times 128 head_dim times 131072 tokens times 2 bytes equals about 171 GB for a single request. A 70B model in FP16 weighs roughly 140 GB. So at 128k context the cache for one request is already larger than the entire model. That is the punchline. At long context, memory is consumed by the cache, not the weights. Adding a second concurrent user means another 171 GB, which is why production systems work hard to shrink the cache.

The KV cache stores one key and one value per token, per layer, per KV head. Its size is the product of every dimension it spans. The formula is: $$\text{bytes} = 2 \cdot L \cdot H_{kv} \cdot d_h \cdot T \cdot b \cdot \text{batch}$$ The leading 2 counts both K and V. Plug in the given numbers. L is 80. KV heads is 64. Head dim is 128. Sequence length is 131072. Bytes per element is 2 for FP16. Batch is 1. That product is about 171 GB for one request. A 70B model in FP16 is roughly 140 GB. So the cache at 128k context already exceeds the weights themselves. This is the key insight. At long context, KV memory is the binding constraint, not parameter count. The model fits; the cache does not. Every concurrent user adds another 171 GB, so throughput is capped by HBM long before compute saturates. Mitigations attack one factor each. Grouped-query attention shrinks the KV head count. KV quantization cuts the bytes per element. Paged attention removes the fragmentation waste. Together they make 128k contexts economically servable.