Grouped-Query Attention (GQA)
Also known as: Grouped Query Attention, GQA
Compromise between MHA and MQA: query heads share KV heads in groups, cutting KV cache by 4-8x.
An attention variant where multiple query heads share a single set of key/value heads. Sits between multi-head attention (one KV per query) and multi-query attention (one KV total), trading quality for KV-cache size.
In practice
Powers LLaMA 2/3 70B and other large open models. Critical for serving cost: the KV cache shrinks proportionally.
How it compares
GQA reduces the KV cache size by sharing; the KV cache itself is what gets cached during inference.
Related topics
Questions that mention this term
Related terms
API LLM
An LLM accessed through a provider API: pay per token, get the frontier model, hand over ops.
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Beam Search
Keep the K best partial sequences at each step; deterministic, breadth-first decoding.
Causal Mask
Attention mask that hides future tokens so each position can only see itself and prior tokens.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
GGUF
Self-contained binary format for quantized LLMs; the standard for llama.cpp / Ollama / LM Studio.