Multi-Head Attention (MHA)
Also known as: MHA, Multi-headed attention
Run several attention heads in parallel with different projections, then concat. This captures multiple relationship types per layer.
A self-attention variant that runs several attention operations in parallel with different learned projections, then concatenates the results. Lets the model attend to different relationship patterns simultaneously.
In practice
Standard attention variant in transformers. GQA and MQA are direct optimizations of it, so knowing MHA first makes those click instantly.
How it compares
MHA gives each query head its own KV; GQA shares KV across groups of query heads.
Related topics
Questions that mention this term
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Causal Mask
Attention mask that hides future tokens so each position can only see itself and prior tokens.
Context Window
The max number of tokens a model can attend to at once.
Decoder-Only
Single autoregressive transformer stack: the shape of every modern frontier LLM.
Encoder-Decoder
Transformer with separate encoder + decoder stacks; strong for translation and structured seq2seq tasks.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.