These four variants exist because of one production problem: the KV cache is huge. Each variant reduces it differently.
**MHA** is the baseline. The input is projected into N query, N key, and N value heads. Every query head has its own K and V. Best quality, biggest cache.
**MQA** is the extreme opposite. Keep N query heads but reduce K, V to ONE shared head. Cache is N times smaller. Quality drops measurably. Used in PaLM and original Falcon.
**GQA** is the compromise. Split the N query heads into G groups. Each group shares one K, V projection. Cache shrinks by N/G. Llama-2/3 use 64 query heads with G=8 → 8x cache savings at near-MHA quality.
**MLA** (DeepSeek-V2 2024) takes a different approach. Compress K and V along the head dimension via a low rank projection into a small latent (~dim 512). Only the latent is cached; per-head K, V are reconstructed at compute time. Cache footprint can be smaller than MQA's, with MHA-level quality.
Progression: MHA → MQA → GQA → MLA, each step is the industry's iterative attack on the KV cache bottleneck.
These four variants sit on a spectrum of how aggressively K and V are shared across query heads.
- **MHA**: each query head has its own K, V. Cache `2 · L · N · d_head · T · b · bytes`. Baseline quality, baseline cache.
- **MQA**: all N query heads share ONE K, V. Cache shrinks by N. Quality drops measurably (PaLM, Falcon-1, StarCoder).
- **GQA**: partition query heads into G groups, each with its own K, V. Cache shrinks by N/G. Llama 4, Llama-3, Mistral, Qwen 3 all ship G=8, sweet spot at 8x savings with near-MHA quality.
- **MLA** (DeepSeek-V2 2024): compress hidden state to a small latent `c_t` (dim ~512), store ONLY the latent per token. Per-head K, V are reconstructed at compute time via absorbed matmuls. Cache can be smaller than MQA at MHA-level quality.
Production picture in 2026: GQA dominates new releases (Llama 4 Maverick, Mistral Large 3, Qwen 3); MLA is the cutting edge in the DeepSeek V2/V3/V4 family; MQA is mostly historical; MHA persists where cache isn't the binding constraint.
Quick reference
TL;DR
MHA: per-head K/V. MQA: one shared K/V (max compression, quality hit). GQA: K/V shared in groups (Llama-2/3 sweet spot). MLA: low rank latent K/V (DeepSeek-V2/V3, smaller than MQA at MHA quality).
Imagine a team of detectives (query heads) each needing a copy of the case file (K, V). MHA gives each detective their own file, best for the work but lots of storage. MQA gives the whole team one shared file, saves space but everyone's investigation is more similar. GQA breaks the team into small groups and gives one file per group, balanced. MLA compresses the file into a shorthand notebook that the team decodes when needed, smallest storage of all.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tiny models under 1B parameters where KV cache fits comfortably in cache memory At this scale the KV cache is megabytes, not gigabytes; the quality cost of MQA or the engineering cost of MLA outweighs the negligible memory win.
Encoder-only models like BERT or sentence transformer encoders used for embeddings There is no autoregressive decode loop, so no KV cache persists across steps. The whole optimization target these variants address does not exist for encoders.
Retrieval or embedding inference where attention compute dominates, not KV memory Short, fixed-length passages mean cache size is bounded and trivial; latency is dominated by the attention matmul, which these variants do not speed up materially.
Very short context chat workloads (under ~512 tokens) on small models Cache grows linearly with T; at T ~ 512 on a sub-7B model, full MHA cache fits in HBM with room to spare and you keep maximum representational capacity.
Research probes that need per-head K/V interpretability or attention-head ablations MQA collapses all heads to one K/V and MLA mixes heads through a shared latent, so per-head causal interventions become ambiguous or impossible.
Quick scan · 2–4 min
Everything important, quickly.
Position each on the sharing spectrum, quote cache-savings ratios, name production users, explain GQA uptraining, and articulate MLA's structural difference (latent compression vs head sharing).
Side-by-side
Variant
K, V heads per layer (vs N query heads)
Cache savings vs MHA
Quality vs MHA
Notable users
MHA
N
1x (baseline)
Baseline
GPT-2/3, Llama-1, BERT
MQA
1
Nx
Drops on some tasks
PaLM, Falcon-1, StarCoder
GQA
G (typically N/8)
N/G x (~8x at G=8)
Near-MHA
Llama-2/3, Mistral, Qwen
MLA
Latent of dim d_c (~512), no head count
≈ 10-20x (depends on d_c)
Matches MHA per DeepSeek
DeepSeek-V2/V3
Real-world examples
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does Ainslie et al. 2023 uptrain an existing MHA checkpoint to GQA?
Sign in or go Pro to see the approach for this follow-up.
QWhy doesn't MLA destroy too much information by compressing K, V to a small latent?
Sign in or go Pro to see the approach for this follow-up.
QWhy is MLA still rare outside DeepSeek?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Confusing MQA with GQA. MQA has exactly one shared K, V across ALL query heads; GQA has multiple K, V groups (more than 1 but fewer than num_query_heads).