Define GQA in transformer attention · Zenaique

Attempt it

Your answer

Quick reference

TL;DR

GQA = Grouped-Query Attention: Q heads are split into G groups that share one K and one V head per group, shrinking the KV cache by the group factor.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine an office where every analyst (a query) has a personal filing cabinet (keys and values). Reading from twenty cabinets every time you decode a token is slow and expensive. Picture grouping the analysts into teams of eight, where each team shares a single cabinet. The analysts still bring their own questions to the table, but the lookup happens against one shared filing cabinet per team. You read far less from the cabinet room each step, and the answers stay almost as good as before. GQA is that shared-cabinet trick for attention, and it is why modern open-weights chatbots feel fast on the same hardware.

Key concepts

Grouped-query attention Attention KV cache Multi-query attention

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: acronym expansion + Q-side untouched, K and V grouped + KV cache formula + Llama 3.1 and Mistral as canonical adopters + the G = 8 sweet spot.

Side-by-side

Variant	KV head count	KV cache size	Quality vs MHA
MHA	n_heads	Full	Baseline
GQA (G = 8)	G = 8	Roughly 1/4 of MHA at 32 heads	Within a fraction of a percent
MQA	1	1 / n_heads	Measurable regression on harder tasks

Real-world examples

Real products, models, and research that use this idea.

Llama 3.1 8B ships GQA with 32 query heads and 8 KV heads, the canonical 4:1 ratio used across the Llama 3 family.
Mistral Large 3 uses GQA at group size 8 to keep the KV cache small enough for long-context serving on H100 fleets.
Qwen 3.5 adopts GQA across its open-weights lineup, matching the Llama-style 8-way grouping.
Llama 4 Scout and Maverick keep GQA in their attention blocks even within the MoE routing layers.
Gemma 4 uses GQA so on-device deployments can run longer contexts within tight memory budgets.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy did the field converge on group size 8 instead of 4 or 16?

Sign in or go Pro to see the approach for this follow-up.

QHow does GQA interact with tensor parallelism across multiple GPUs?

Sign in or go Pro to see the approach for this follow-up.

QWhen is GQA the wrong choice and you would prefer MLA or MQA instead?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Saying GQA reduces FLOPs. The savings are bandwidth and KV-cache memory, not compute. Q projections still run per head.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.