Multi-Query Attention

Expand MQA and state its KV-cache cost tradeoff

Define GQA in transformer attention

What do you gain and lose by tying K and V to one projection?

PremiumWhy does MQA underperform…

What does MQA stand for, and what do the heads share?

What does GQA stand for, and describe its KV sharing pattern?

Why is decode phase attention memory bandwidth bound when the model generates one token at a time?

Match MLA, GQA and MQA to their mechanisms, cache reduction and quality posture

Which statement most accurately captures why GQA, not MQA, became the production default?

Match each attention variant to its KV-cache property and a production user

Compare MHA, MQA, GQA, and MLA. What production tradeoff does each one address, and which models use it?

PremiumMatch each attention variant…