Multi-Head Attention (MHA)
Also known as: MHA, Multi-headed attention
Run several attention heads in parallel with different projections, then concat. This captures multiple relationship types per layer.
A self-attention variant that runs several attention operations in parallel with different learned projections, then concatenates the results. Lets the model attend to different relationship patterns simultaneously.
In practice
Standard attention variant in transformers. GQA and MQA are direct optimizations of it, so knowing MHA first makes those click instantly.
How it compares
MHA gives each query head its own KV; GQA shares KV across groups of query heads.
Related topics
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Transformer
The attention-only neural architecture behind GPT, Claude, Gemini, and almost every modern LLM.
Context Window
The max number of tokens a model can attend to at once.
Mixture of Experts (MoE)
Scale model capacity by routing each token to a small subset of expert MLPs instead of using all of them.
FlashAttention
A memory-aware attention kernel that's 2-4x faster than vanilla, with identical math.
RoPE (Rotary Position Embedding)
Position info injected by rotating Q and K vectors, easy to extend to longer contexts.