Decode the acronym MHA in the transformer attention context.
MHA stands for Multi-Head Attention, the Vaswani 2017 baseline where h independent (Q, K, V) heads run in parallel inside one attention layer.
Picture a panel of expert reviewers reading the same document. Each reviewer has a different angle: one tracks grammar, one tracks logic, one tracks tone, one tracks references. They all read the same text at the same time, write their notes independently, and the editor then merges the notes into a final report. MHA is that panel inside one step of a transformer. Several reviewers (called heads) work on the same sentence at the same time, each with their own private set of reading glasses, each picking up on a different kind of relationship between words. At the end, their notes are stitched together and combined into one summary the model uses to keep thinking.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multi-Head Attention (MHA) is the baseline attention pattern that defined the transformer architecture in 2017, and the one every subsequent variant is compared against. Knowing what each letter stands for is table stakes; understanding why the multi-head structure exists, what it does and does not change, and how MQA, GQA, and MLA descend from it is the difference between a surface acronym recall and a working mental model of modern attention.
The acronym is simple: Multi-Head Attention. The mechanism is a parallel split: one attention layer runs h independent (Q, K, V) projection sets, computes scaled dot-product attention per head, concatenates the results, and mixes them through the output projection W_O.
The deeper question is why parallel heads matter. The single most counterintuitive fact about MHA is that it does NOT add parameters over a single-head attention at the same d_model. The Q, K, V, O projections are each d_model x d_model total, sliced into h heads of width d_head = d_model / h. What you gain is parallel specialization, not capacity.
This deep dive walks the mechanism, the parameter accounting, why specialization happens, and how the GQA, MQA, MLA family descends from MHA by attacking only the K, V side.
The mechanism, formal
Given input X of shape (batch, seq, d_model), the MHA layer computes:
Project to h heads
Q = X W_Q, K = X W_K, V = X W_V where each W is d_model x d_model. Reshape into (batch, h, seq, d_head) where d_head = d_model / h.
Per-head attention
For each head i, compute head_i = softmax(Q_i K_i^T / sqrt(d_head)) V_i. This is exactly the scaled dot-product attention formula:
with the heads running in parallel as a batched matmul.
Concat and project
Concatenate the h per-head outputs back to width d_model, then apply the output projection:
W_O is d_model x d_model and mixes head outputs adaptively. Without it, the output would just be a concatenation of disjoint per-head slices.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- The original Transformer in Vaswani et al. 2017 used MHA with h=8 heads at d_model=512.
- BERT-base used MHA with h=12 heads at d_model=768; BERT-large used h=16 at d_model=1024.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the output projection W_O matter rather than just concatenating heads?
Concatenation alone gives a fixed per-head slice of the output. W_O is a learned d_model x d_model matrix that mixes head outputs adaptively, letting the model decide how much each head contributes to each output channel.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reading MHA as 'masked head attention' or thinking the h heads increase parameter count. Heads partition the existing d_model budget; the M stands for Multi, not Masked.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.