Match each attention variant to its KV-cache memory reduction strategy.

These four variants exist because of one production problem: the KV cache is huge. Each variant reduces it differently. **MHA** is the baseline. The input is projected into N query, N key, and N value heads. Every query head has its own K and V. Best quality, biggest cache. **MQA** is the extreme opposite. Keep N query heads but reduce K, V to ONE shared head. Cache is N times smaller. Quality drops measurably. Used in PaLM and original Falcon. **GQA** is the compromise. Split the N query heads into G groups. Each group shares one K, V projection. Cache shrinks by N/G. Llama-2/3 use 64 query heads with G=8 → 8x cache savings at near-MHA quality. **MLA** (DeepSeek-V2 2024) takes a different approach. Compress K and V along the head dimension via a low rank projection into a small latent (~dim 512). Only the latent is cached; per-head K, V are reconstructed at compute time. Cache footprint can be smaller than MQA's, with MHA-level quality. Progression: MHA → MQA → GQA → MLA, each step is the industry's iterative attack on the KV cache bottleneck.

These four variants sit on a spectrum of how aggressively K and V are shared across query heads. - **MHA**: each query head has its own K, V. Cache `2 · L · N · d_head · T · b · bytes`. Baseline quality, baseline cache. - **MQA**: all N query heads share ONE K, V. Cache shrinks by N. Quality drops measurably (PaLM, Falcon-1, StarCoder). - **GQA**: partition query heads into G groups, each with its own K, V. Cache shrinks by N/G. Llama 4, Llama-3, Mistral, Qwen 3 all ship G=8, sweet spot at 8x savings with near-MHA quality. - **MLA** (DeepSeek-V2 2024): compress hidden state to a small latent `c_t` (dim ~512), store ONLY the latent per token. Per-head K, V are reconstructed at compute time via absorbed matmuls. Cache can be smaller than MQA at MHA-level quality. Production picture in 2026: GQA dominates new releases (Llama 4 Maverick, Mistral Large 3, Qwen 3); MLA is the cutting edge in the DeepSeek V2/V3/V4 family; MQA is mostly historical; MHA persists where cache isn't the binding constraint.