Llama-2 70B uses 64 query heads, how many KV heads does it actually keep?

**Llama-2 70B uses 8 KV heads** for its 64 query heads. The pattern is grouped-query attention (GQA) with group size 8: query heads partition into 8 groups, and each group shares one K head and one V head. The motivation is the KV cache. During autoregressive decode, the model stores K and V for every cached token per head per layer. With 80 layers and long context, plain multi-head attention at 64 KV heads would need roughly 21 GB of KV cache per concurrent request. GQA at 8 KV heads cuts that to 2.7 GB, an 8x reduction. The quality cost is small. Meta's ablations during the Llama 2 design showed GQA-8 essentially matched plain MHA on standard benchmarks, while the 8x memory saving made interactive long-context serving practical at 70B scale. The smaller Llama-2 variants (7B and 13B) kept plain MHA in the original release. At those sizes the KV cache was not the binding inference constraint, and the small quality safety margin of MHA was worth keeping. The 70B was where GQA's economics dominated.

Llama-2 70B uses **8 KV heads** for its 64 query heads. This is GQA with group size 8: each KV head is shared by 8 query heads. The relationship between configurations: - 64 KV heads = plain MHA (one per query head). What GPT-2 and Llama-2 7B/13B use. - 32 KV heads = GQA-2 (pairs of queries share). Not what Llama-2 picked. - **8 KV heads = GQA-8**. What Llama-2 70B picked. - 1 KV head = MQA (all queries share). Used in PaLM and some Gemini variants, not Llama-2. The choice was driven by inference memory. The KV cache for a 70B model at 8k context with plain MHA is roughly 21 GB per concurrent request. With GQA-8 it drops to 2.7 GB. On an H100 with ~20 GB available per shard for KV cache after weights and activations, that is the difference between serving 1 concurrent request and serving 7-8. Meta's ablations showed GQA-8 matched plain MHA within noise on standard benchmarks, while G = 16 or higher started showing a perplexity hit. G = 8 was the sweet spot: largest free saving, smallest measured quality risk. This design is now baseline for production LLMs above 30B parameters.