MQA, spell out the acronym and name exactly what gets shared across heads

MQA stands for **Multi-Query Attention**. It is a variant of standard multi-head attention designed to shrink the KV cache, which is the dominant memory cost during long-context inference. In standard multi-head attention (MHA), every attention layer has h heads, and each head has its own Q, its own K, and its own V projection matrices. The KV cache therefore stores h separate K and V tensors per layer. MQA keeps the query side multi-headed (h independent Q heads per layer) but collapses the K and V side to a single shared head. All h query heads attend to the same K and V tensors. The cache shrinks from h K and V slices per layer to just one, a factor-of-h memory reduction. The tradeoff is a measurable quality drop relative to MHA, because all heads now look at the same K, V subspace and lose some of the parallel specialization that multi-head buys. GQA was introduced as a middle ground that shares K, V within small groups of query heads, capturing most of the cache savings with much less quality loss. MQA was the dominant choice in Google's PaLM and several Gemini variants; Llama-2 70B and most 2026 production decoders prefer GQA.

MQA expands to **Multi-Query Attention**. The acronym is precise: queries stay multi-headed; keys and values collapse to a single shared slice per layer. Mechanically, a layer with h heads keeps `W_Q` as a `(d_model, h * d_head)` projection (h independent query heads) but reduces `W_K` and `W_V` to `(d_model, d_head)` (one K head, one V head). At attention time, all h query heads dot against the same K and softmax over the same V. The payoff is that the KV cache stores just one K and one V slice per layer instead of h. For a 70B-class model with h=64 heads, this is a 64x reduction in cache memory, which makes long-context serving dramatically cheaper. The cost is a measurable perplexity bump relative to MHA. Forcing all query heads to read from the same K, V subspace throws away most of the per-head specialization in the lookup direction. Different heads can still ask different questions (different Qs), but they all index into the same memory, which limits how independently they can attend. GQA splits the difference by sharing K, V within groups of query heads (Llama 4 Maverick uses 8 KV heads per 64 query heads). Production decoders in 2026 have largely converged on GQA, MLA, or hybrids; pure MQA appears mostly in older Google models and some inference-optimized variants.