Match MLA, GQA and MQA to their mechanisms, cache reduction and quality posture

Attempt it

MLA (DeepSeek)

GQA (Llama-3 style)

MQA (PaLM-style)

Reconstruction cost

Empirical 2026 winner

Quick reference

TL;DR

MQA shares one KV head, GQA shares KV heads per group, and MLA stores a learned low-rank latent that reconstructs K and V on demand: three ways to shrink the KV cache.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a library where every reader carries a full set of reference notes. That is multi-head attention: lots of duplicated notes, huge shelves. MQA says everyone share one single set of notes, cheapest but coarse. GQA says split readers into a few teams and give each team one shared set, a sensible middle ground. MLA says do not store the bulky notes at all, store a tiny compressed summary, then unpack the details only when a reader actually needs them. The summary fits in a drawer instead of a shelf. Unpacking costs a little extra work each time, but the storage savings are enormous, which is exactly the trade modern serving systems want.

Key concepts

KV cache Multi-head latent attention Grouped-query attention Multi-query attention

Quick scan · 2–4 min

Everything important, quickly.

4 min: place all three on the compression spectrum, contrast store and share (MQA, GQA) with compress and reconstruct (MLA), cover MLA's up-projection and decoupled RoPE, then map each to its 2026 model.

Side-by-side

Method	What is stored	Cache reduction	Quality posture	Runtime catch
MHA	Full K and V per head	1x baseline	Reference quality	Largest cache, decode is bandwidth-bound
MQA	One shared K and V head	n_heads x	Regresses on hard tasks	Coarsest single history view
GQA	One K and V head per group	G x (G=8 typical)	Near full quality	None beyond standard attention
MLA	Low-rank latent per token	~10x	Matches GQA	Up-projection matmul, decoupled RoPE

Real-world examples

Real products, models, and research that use this idea.

DeepSeek-V3 ships Multi-head Latent Attention to cut per-request cache about tenfold versus full multi-head attention while matching GQA-level quality.
Llama-3 and Llama 4 use Grouped-Query Attention with 8 KV groups, the de facto default across most open frontier models in 2026.
Multi-Query Attention, the single-KV-head extreme, survives in 2026 mainly in latency-critical edge stacks; GQA refined it for better quality retention.
vLLM and SGLang implement GQA-aware paged KV cache so the per-group sharing translates directly into larger serving batches.
DeepSeek-V3's MLA uses a decoupled RoPE path so rotary positions stay consistent under the low-rank latent projection.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does MQA regress on hard reasoning tasks while GQA largely does not?

Sign in or go Pro to see the approach for this follow-up.

QHow does MLA reconstruct K and V, and how is the reconstruction cost hidden?

Sign in or go Pro to see the approach for this follow-up.

QWhy does MLA need a decoupled RoPE path?

Sign in or go Pro to see the approach for this follow-up.

QWhen would you still pick GQA over MLA despite the smaller cache win?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.