GQA, decode the acronym and describe its KV sharing pattern

**GQA** stands for **Grouped-Query Attention**. The structural idea is straightforward: query heads partition into G groups, and each group shares a single K head and a single V head. Concretely, if you have 64 query heads and choose group size 8: - Heads 0-7 share KV head 0. - Heads 8-15 share KV head 1. - And so on through KV head 7. You end up with 64 query heads but only 8 K and 8 V heads. GQA sits between two extremes: - **MHA (multi-head attention)**: G = 1 (every query head has its own KV). Maximum quality, maximum KV cache memory. - **MQA (multi-query attention)**: G = num_heads (all queries share one KV). Minimum KV cache, but a measurable quality drop. GQA interpolates. Llama-2 70B chose 64 query heads on 8 KV heads. The KV cache shrinks 8x relative to plain MHA, and Meta's ablations showed the quality loss versus full MHA is small enough to be in the noise on most benchmarks. The reason this matters is the KV cache. During autoregressive decode, the model stores K and V for every cached token, per head, per layer. GQA divides that storage cost by the group size.

**GQA, Grouped-Query Attention**, partitions query heads into G groups and gives each group a single shared K head and V head. With `num_heads` query heads and `num_kv_heads` KV heads, the group size is `num_heads / num_kv_heads`. Llama-2 70B is the canonical example: 64 query heads, 8 KV heads, group size 8. Inside each group, the 8 query heads compute different attention scores against the same K, but the V they multiply is also shared. The output projection still mixes across heads as usual, so the model retains most of the expressive capacity of full MHA. GQA interpolates between two known designs: - **MHA**: G = 1 (one KV head per query head). Maximum quality, maximum KV cache. - **MQA**: G = num_heads (one KV head for everyone). Minimum KV cache, measurable quality drop. The key win is the KV cache. During decode, the cache stores K and V for every past token per head per layer. A 70B model at 8k context with plain MHA needs roughly 21 GB of KV cache per concurrent request. GQA at group size 8 cuts that to 2.6 GB. At small batch sizes (interactive serving), that is the difference between serving 1 request per H100 and serving 8. The quality cost is small. Meta's ablations show GQA-8 matches MHA within noise on standard benchmarks. The pattern is now standard for production LLMs above 30B parameters.