Which statement most accurately captures why GQA, not MQA, became the production default?

Attempt it

Quick reference

TL;DR

GQA gives a tunable knob: share K and V across groups of query heads to shrink the KV cache like MQA, while keeping enough KV heads to hold MHA-level quality.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a classroom where every student (a query head) needs to consult reference notes before answering. Multi-head attention gives each student a personal, hand-written copy of the notes, which is accurate but eats a huge amount of paper. Multi-query attention hands the whole class a single shared copy, saving paper but causing crowding and mistakes when the material gets hard. Grouped-query attention splits the class into small groups, say eight students per shared copy. You still save most of the paper, but each group's notes stay specialized enough that answers barely suffer. That middle setting is why modern language models pick it: nearly the savings of one shared copy, nearly the quality of personal copies.

Key concepts

Grouped-query attention Multi-query attention Multi-head attention KV cache

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: define MHA, MQA, GQA by KV head count, then the memory-bandwidth axis, the group-size knob and endpoints, why eight, uptraining, and where MLA fits.

Real-world examples

Real products, models, and research that use this idea.

Llama 3.1 70B ships GQA with eight KV heads, the canonical production setting that balances cache size against quality.
Mistral Large 3 uses grouped-query attention so long-context decode stays within HBM budgets on a single node.
DeepSeek V4 moves past GQA to Multi-head Latent Attention, compressing K and V to a low-rank latent for an even smaller cache.
vLLM and SGLang both assume GQA-style KV layouts when sizing paged-attention blocks for open models in 2026.
Gemini 3.1 Pro and Claude Opus 4.7 serving stacks rely on grouped or latent KV schemes to make multi-request long-context inference economical.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does sharing KV heads barely change compute even though it cuts memory?

Sign in or go Pro to see the approach for this follow-up.

QHow was GQA bootstrapped from existing MHA checkpoints rather than trained fresh?

Sign in or go Pro to see the approach for this follow-up.

QWhere does Multi-head Latent Attention sit relative to GQA on the same axis?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming GQA wins on compute or parameter count. The win is KV-cache bytes and bandwidth; the query side matmul is unchanged, and only the K and V projections shrink.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.

Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints