Grouped-Query Attention (GQA)
Also known as: Grouped Query Attention, GQA
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.
An attention variant where multiple query heads share a single set of key/value heads. Sits between multi-head attention (one KV per query) and multi-query attention (one KV total), trading quality for KV-cache size.
In practice
Powers LLaMA 2/3 70B and other large open models. Critical for serving cost: the KV cache shrinks proportionally.
How it compares
GQA reduces the KV cache size by sharing; the KV cache itself is what gets cached during inference.
Related topics
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Transformer
The attention-only neural architecture behind GPT, Claude, Gemini, and almost every modern LLM.
KV Cache
Cache attention's K and V tensors per layer so each new token doesn't re-process every prior token.
Temperature
Sampling knob (low = focused, high = diverse), applied to the logits before softmax.
Quantization
Run the model at lower numerical precision to save memory and accelerate inference.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.