MHA stands for Multi-Head Attention, the baseline attention pattern from Vaswani 2017 where every attention layer runs h independent (Q, K, V) projection sets in parallel, computes scaled dot-product attention per head, concatenates the per-head context vectors back into a d_model-wide vector, and finishes with the output projection W_O. Each head has its own K and its own V (no sharing across heads). This is the configuration MQA and GQA are KV-cache-saving variants of: MQA collapses to one shared K, V across all query heads; GQA shares K, V within groups of query heads.
MHA stands for Multi-Head Attention. It is the baseline attention pattern introduced in the 2017 'Attention Is All You Need' paper, and the one every newer variant (MQA, GQA, MLA) is compared against.
The idea is to run h independent attention computations in parallel inside the same attention layer. Each one is called a 'head'. Every head has its own Q, K, and V projection matrices, but those matrices are smaller than they would be in a single-head layer: each head works in a smaller `d_head = d_model / h` subspace.
For a single token, every head independently computes scaled dot-product attention over the sequence, producing its own context vector. Those h context vectors are then concatenated end to end (so the total width is back to d_model) and passed through one final output projection `W_O` to mix the heads' outputs.
The payoff is parallel specialization. Different heads tend to learn to attend to different relations (syntactic, positional, coreference, etc.), giving the layer a richer representation than a single head of the same total width.
MHA is Multi-Head Attention, the Vaswani 2017 baseline. A single attention layer splits its `d_model`-wide computation into h parallel heads, each running scaled dot-product attention in a `d_head = d_model / h` subspace.
Mechanically: project the input into h independent (Q, K, V) triples, compute attention per head, concatenate the h output vectors back to width `d_model`, then apply the output projection `W_O`.
The multi-head structure does not add parameters versus a single d_model attention; the Q, K, V projections are still `d_model x d_model` total, just sliced into h blocks of `d_head` width. What it buys is parallel specialization, different heads learn different relations (one tracks adjacency, one tracks coreference, one tracks long range syntactic links) inside the same parameter budget.
MHA is the structure that MQA, GQA, and MLA modify. MQA shares one K, V across all query heads. GQA shares K, V within small groups of query heads (Llama 4 Maverick uses 8 KV heads per 64 query heads). MLA in DeepSeek V4 compresses K, V into a low rank latent. All three preserve the multi-head query side and only attack the K, V side, which is what shrinks the KV cache without losing query-side specialization.
Quick reference
TL;DR
MHA stands for Multi-Head Attention, the Vaswani 2017 baseline where h independent (Q, K, V) heads run in parallel inside one attention layer.
Picture a panel of expert reviewers reading the same document. Each reviewer has a different angle: one tracks grammar, one tracks logic, one tracks tone, one tracks references. They all read the same text at the same time, write their notes independently, and the editor then merges the notes into a final report. MHA is that panel inside one step of a transformer. Several reviewers (called heads) work on the same sentence at the same time, each with their own private set of reading glasses, each picking up on a different kind of relationship between words. At the end, their notes are stitched together and combined into one summary the model uses to keep thinking.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Long-context production decoders where KV cache dominates HBM Pure MHA pays full cache cost per query head; GQA or MLA cuts the dominant memory term with minimal quality loss.
Edge or mobile inference with tight memory budgets MQA or GQA reduces cache and weight memory, often enabling deployment that pure MHA cannot fit.
Small models where head specialization adds little At very small d_model, splitting into many heads gives each head too few dimensions to learn distinct relations, and a wider single-head variant may train more stably.
Architectures replacing attention with a different mixer State-space models like Mamba and linear attention variants do not use the multi-head structure at all, so MHA is not the relevant baseline.
Quick scan · 2–4 min
Everything important, quickly.
3m: what MHA stands for, the parallel-head mechanism, why parameters are unchanged vs single-head, parallel specialization across heads, and the relationship to MQA, GQA, MLA.
Real-world examples
Real products, models, and research that use this idea.
1The original Transformer in Vaswani et al. 2017 used MHA with h=8 heads at d_model=512.
2BERT-base used MHA with h=12 heads at d_model=768; BERT-large used h=16 at d_model=1024.
3GPT-2 and GPT-3 used MHA throughout, before the field shifted to GQA in larger production decoders.
4Llama 4 Maverick and Mistral Large 3 use GQA, the production successor to MHA that keeps multi-head queries but shares K and V.
5DeepSeek V4 uses MLA, another MHA descendant that compresses K and V into a low rank latent while preserving multi-head queries.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the output projection W_O matter rather than just concatenating heads?
Sign in or go Pro to see the approach for this follow-up.
QIf MHA has the same parameter count as single-head, why is the multi-head version empirically better?
Sign in or go Pro to see the approach for this follow-up.
QHow would you describe the relationship between MHA and GQA in one sentence?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reading MHA as 'masked head attention' or thinking the h heads increase parameter count. Heads partition the existing d_model budget; the M stands for Multi, not Masked.