For a 64-Q-head model with GQA group size G=8, how many K/V heads exist and what is the cache reduction vs MHA? · Zenaique

Attempt it

A transformer has 64 query heads. It uses GQA with group size G=8. How many K and V heads does it have? What is the KV-cache reduction factor relative to plain MHA at the same head_dim and sequence length? Does it cost extra FLOPs at inference time?

Free · 2 AI evals / day

Quick reference

TL;DR

K and V heads equal 64/8 = 8, so the KV cache shrinks by exactly the group size G = 8x versus MHA, with zero extra inference FLOPs.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture 64 readers in a library, each needing a reference book. Under plain attention every reader gets a private copy, so the shelf holds 64 books. GQA arranges the readers into 8 teams of 8, and each team shares one book. Now the shelf holds only 8 books, eight times less storage. The readers still read independently and ask their own questions, so no reader does extra work. The only change is that teammates look at the same shared book instead of personal copies. You keep the same number of readers and the same reading effort, but you slash the shelf space. That shelf is the KV cache, and shrinking it is what lets you serve long contexts without running out of memory.

Key concepts

Grouped-query attention KV cache Multi-query attention

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

3 min: do the 64/8 division, derive the cache reduction from the formula, confirm no FLOP penalty, then explain the HBM bandwidth win on decode.

Real-world examples

Real products, models, and research that use this idea.

Llama 3.1 70B ships GQA with 8 KV heads against 64 query heads, the textbook ratio in this question.
Mistral Large 3 uses GQA to keep its KV cache small enough for long-context serving on a single node.
vLLM and SGLang implement GQA-aware paged attention so the shared KV heads pack densely in HBM.
DeepSeek V4 went further with Multi-head Latent Attention, compressing KV beyond what an 8-group GQA achieves.
TensorRT-LLM on H100 and B200 exploits the smaller GQA KV read to raise decode throughput on Llama-class models.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the reduction factor equal the group size and not the head dim ratio?

Sign in or go Pro to see the approach for this follow-up.

QWhy is there genuinely no extra inference FLOP cost despite sharing K and V?

Sign in or go Pro to see the approach for this follow-up.

QHow does GQA differ from MQA, and when would you pick one over the other?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming GQA reduces query heads or adds inference FLOPs. Q heads stay at 64 and arithmetic is unchanged; only the count of cached K and V heads drops.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.

Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints