GQA, decode the acronym and describe its KV sharing pattern

Attempt it

GQA stands for Grouped-Query Attention. Query heads are partitioned into G groups; each group shares a single K head and a single V head. So if you have 64 query heads and group size 8, you end up with 64 / 8 = 8 KV heads. GQA interpolates between full multi-head attention (G = 1, every query head gets its own KV) and multi-query attention (G = num_heads, all queries share one KV). Llama-2 70B uses 64 query heads with 8 KV heads, an 8x KV-cache reduction relative to MHA while losing very little quality versus full MHA on the original ablations.

Your answer

Quick reference

TL;DR

GQA = Grouped-Query Attention. Query heads partition into G groups; each group shares one K and one V head. Llama-2 70B uses 64 query heads on 8 KV heads.

Memory aid

Easy to grasp

Imagine 64 people on a quiz team, each holding a different question. In regular multi-head attention, every person gets a personal librarian and a personal bookshelf. That is expensive: 64 librarians, 64 shelves. Grouped-query attention says: split the 64 people into 8 groups of 8, and each group shares one librarian and one shelf. The 8 people in a group ask different questions, but they all consult the same librarian for answers. You go from 64 shelves down to 8, an 8x storage saving, and the team's quiz scores barely drop because the librarians were duplicating each other's work anyway.

Key concepts

Grouped-query attention Multi-head attention Multi-query attention KV cache

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

60s: expand the acronym, describe the group partitioning, name the MHA-MQA slider with GQA in between, give Llama-2 70B's specific config (64 Q, 8 KV), state the KV cache reduction (8x), mention the quality cost is small per Meta's ablations.

Real-world examples

Real products, models, and research that use this idea.

Llama-2 70B: 64 query heads, 8 KV heads, group size 8. The original GQA production deployment.
Llama 4 Maverick and Mistral Large 3 inherit Llama-2's GQA pattern for KV cache reduction at scale.
Qwen 3.5 series uses GQA with model size dependent group sizes (smaller groups on larger variants).
Gemma 4 uses GQA across its model family, with the group size tuned per size class.
DeepSeek V4 supersedes GQA with MLA (multi-head latent attention), which compresses KV further into a learned latent representation.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does GQA cost so little quality compared to plain MHA?

QWhy not just use MQA and get the maximum KV cache saving?

QHow does GQA interact with FlashAttention and Flash-Decoding?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Thinking GQA is just MQA with extra heads. Both share KV across queries, but GQA picks a group size between 1 (MHA) and num_heads (MQA) to balance memory savings against quality.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.