Grouped Query Attention

Spot the GQA configuration error: num_attention_heads=32, num_key_value_heads=7.

Match MLA, GQA and MQA to their mechanisms, cache reduction and quality posture

Compute the KV cache size for a 70B model at 128k context (80 layers, 64 heads, head_dim 128, FP16).

Match each attention variant to its KV-cache property and a production user

For a 64-Q-head model with GQA group size G=8, how many K/V heads exist and what is the cache reduction vs MHA?