Expand MQA and state its KV-cache cost tradeoff

**MQA** stands for **Multi-Query Attention**. It is a variant of attention that aggressively shrinks the KV cache by collapsing every key and value head into a single shared pair. In standard multi-head attention, a layer with 32 Q heads also has 32 K heads and 32 V heads. Each head has its own projection matrix and contributes its own slab to the KV cache. MQA keeps all 32 Q projections (so query diversity is preserved), but it replaces the 32 K and 32 V tensors with one K and one V tensor that every query head reads against. The consequence is a KV cache that is `n_heads` times smaller, which is a massive bandwidth and memory win at inference time. Decode reads vastly less from HBM each step, and long-context deployments fit far more easily on a single GPU. The cost is quality. With a single shared K and V tier, the model loses some ability to route information through head-specific specialization. On harder benchmarks the regression versus MHA is measurable. PaLM and Falcon-40B shipped MQA early; later models like Llama 3.1 and Mistral switched to GQA, which keeps most of the savings without the quality loss.

**MQA**, Multi-Query Attention, is the most aggressive KV-cache reduction in the standard attention family. Each query head keeps its own projection, so Q-side diversity is preserved, but all query heads share a single K head and a single V head per layer. The inference math becomes lopsided in a useful way. The KV cache shrinks by a factor of `n_heads` because there is only one K and one V tensor to store per layer instead of one per head. On a 32-head model that is a 32x reduction in cache memory and KV-cache bandwidth at decode. The quality picture is more complicated. With only one shared K and V tier, different heads can no longer pull information from different stored representations. They all attend to the same keys and aggregate the same values, distinguished only by their query projection. On simpler tasks this rarely matters, but on harder reasoning and long-context evals there is a measurable regression versus MHA. PaLM and Falcon-40B shipped MQA in production, and the savings clearly worked for them. But the field largely moved to GQA, which preserves most of the cache savings (group size 8 is the modern default) while closing nearly all of the quality gap. MQA still appears in tightly memory-constrained settings, but GQA is the 2026 standard.