Expand MQA and state its KV-cache cost tradeoff · Zenaique

Attempt it

MQA stands for Multi-Query Attention. Every Q head still has its own projection, but all Q heads share a single K head and a single V head. This is the most aggressive KV-cache reduction available, the cache shrinks by a factor of n_heads, but it pays a measurable quality cost on harder tasks compared with MHA. PaLM and Falcon-40B shipped MQA before the field largely moved to GQA, which keeps most of the cache savings without the quality regression.

Your answer

Quick reference

TL;DR

MQA = Multi-Query Attention: every Q head keeps its own projection but all heads share one K and one V, shrinking the KV cache by n_heads at a measurable quality cost.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a meeting room with twenty people asking different questions, each consulting their own personal filing cabinet. MQA tears out nineteen of the cabinets and tells everyone to share the one that remains. The questions still vary person to person, but they all consult the same source of information. That makes the room dramatically cheaper to run, because there is only one cabinet to maintain. The catch is that the single shared cabinet has less room to specialize, so subtle answers may drop in quality. MQA is that aggressive sharing trade, useful when you need to squeeze the last byte of memory out of a deployment.

Key concepts

Multi-query attention Attention Grouped-query attention KV cache

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: acronym expansion + Q stays per head, K and V collapse to one + factor of n_heads cache reduction + measurable quality regression + PaLM and Falcon-40B as adopters, modern default now GQA.

Side-by-side

Variant	K and V heads	KV cache size	Quality vs MHA
MHA	n_heads	Full	Baseline
MQA	1	1 / n_heads	Measurable regression on harder tasks
GQA (G = 8)	G = 8	G / n_heads	Within a fraction of a percent

Real-world examples

Real products, models, and research that use this idea.

PaLM, the early Google decoder-only model, used MQA throughout to keep its KV cache small at scale.
Falcon-40B from TII shipped MQA, which let it serve faster decode than equivalently sized MHA models.
Llama 3.1 and Mistral Large 3 dropped MQA in favor of GQA at group size 8, recovering quality with most of the savings.
ChatGLM and several early 7B-class open models adopted MQA to keep decoder KV cache small before GQA became standard.
Some on-device deployments of small Llama variants still use MQA when KV cache must fit in tight phone-class memory budgets.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does collapsing K and V hurt quality more than collapsing Q would?

Sign in or go Pro to see the approach for this follow-up.

QCan you upgrade an MHA checkpoint to MQA without retraining from scratch?

Sign in or go Pro to see the approach for this follow-up.

QWhere does MQA still win over GQA at group size 8?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing MQA with GQA. MQA shares one K and V across the entire layer; GQA shares per group of heads.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.