MQA, spell out the acronym and name exactly what gets shared across heads

Attempt it

MQA stands for Multi-Query Attention. Every query head keeps its own Q projection, so there are still num_heads independent queries per token, but ALL query heads share a SINGLE K head and a SINGLE V head. The KV cache therefore stores just one K and one V slice per layer instead of num_heads of them, cutting cache memory by a factor of num_heads. The cost is a measurable but small quality drop versus full multi-head attention; GQA exists as a middle ground that recovers most of that quality. MQA was the dominant choice in PaLM and several Gemini variants; Llama-2 70B picked GQA instead.

Your answer

Quick reference

TL;DR

MQA is Multi-Query Attention: queries stay multi-headed, but ALL heads share a single K and single V, shrinking KV cache by a factor of num_heads.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a panel discussion where each panelist asks their own questions (the queries) but they all share one big shared notebook of source material (the keys and values). Standard multi-head attention gives each panelist their own private notebook, which is great for diversity but expensive to print and carry. MQA keeps the diverse questioners and gives them one shared notebook, much cheaper to manage at the cost of slightly less specialized lookups.

Key concepts

Multi-query attention KV cache

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3m: what MQA stands for and what stays multi-headed, projection shapes for W_Q vs W_K, W_V, the KV cache reduction factor, why quality drops, and how GQA recovers most of the quality at slightly more cache.

Real-world examples

Real products, models, and research that use this idea.

Google's PaLM used MQA throughout, exploiting TPU memory bandwidth to make the cache shrink the dominant inference win.
Falcon-40B and Falcon-180B used MQA in their initial release, before the field largely moved to GQA.
Several Gemini variants inherited the MQA pattern from PaLM-era Google research.
Inference-optimized Llama-2 retrofits experimented with collapsing trained MHA models to MQA, with measurable quality loss.
GQA replaced MQA as the production default in Llama 4 Maverick, Claude Opus 4.7, and Mistral Large 3.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does Ainslie et al.'s GQA paper describe MQA as too aggressive?

Sign in or go Pro to see the approach for this follow-up.

QHow would you retrofit a trained MHA model to MQA at inference time?

Sign in or go Pro to see the approach for this follow-up.

QWhat does MQA imply for FlashAttention kernels?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying MQA collapses queries too. Only K and V are shared; query heads remain independent. The shared-K-V part is what shrinks the cache; the multi-query part is what preserves query-side specialization.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.