One head owns its own slice of the Q, K, and V projections, a (d_model x d_head) matrix for each of W_Q, W_K, W_V, where d_head = d_model / num_heads. Using those projections it computes its own attention weights softmax(QK^T / sqrt(d_head)) and produces its own d_head-dimensional context vector per token. Heads run fully in parallel and never talk to each other inside the attention layer. Their per-token context vectors are concatenated along the feature axis, then a shared output projection W_O of shape (d_model x d_model) mixes them back into the residual stream.
A single **attention head** is the smallest independent attention computation inside a multi-head attention layer. It owns three things and produces one thing.
What it owns: its own slice of the query, key, and value projections. For a model with `d_model = 4096` and 32 heads, each head gets a head dimension of `d_head = 128`. Inside one head you have three small projection matrices, W_Q, W_K, and W_V, each of shape `(d_model, d_head) = (4096, 128)`.
What it computes: using those projections it produces its own Q, K, V matrices of shape `(seq_len, 128)` per token, then runs scaled dot-product attention:
$$\text{Attn}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$
What it produces: a `d_head = 128` dimensional context vector per token.
All heads run fully in parallel and never look at each other's outputs inside the attention layer. Their per-token outputs get concatenated along the feature axis to width `d_model`, then a shared output projection W_O of shape `(d_model, d_model)` mixes them back into the residual stream.
A single **attention head** is one independent (Q, K, V) computation inside multi-head attention. It owns its three projection matrices, runs its own softmax, and writes out a vector at its head width.
For `d_model = 4096` and `n_heads = 32`, the standard recipe gives `d_head = d_model / n_heads = 128`. Each head owns:
- `W_Q^h` of shape `(d_model, d_head)` = (4096, 128)
- `W_K^h` of shape `(d_model, d_head)` = (4096, 128)
- `W_V^h` of shape `(d_model, d_head)` = (4096, 128)
Given input X of shape (seq_len, d_model), the head computes Q, K, V of shape (seq_len, 128) and then `softmax(QK^T / sqrt(d_head)) V`. The output is a (seq_len, 128) context tensor.
Heads run in parallel. They never communicate inside the attention layer, this is a load-bearing property because it is what makes multi-head attention embarrassingly parallel across GPU lanes. The integration happens after concatenation:
- Concatenate all head outputs along the feature axis: shape becomes `(seq_len, d_model)`.
- Apply the shared output projection `W_O` of shape `(d_model, d_model)`.
- Add to the residual stream.
That output projection is the only place where information from different heads mixes. In implementations the per-head W_Q matrices are usually packed into a single `(d_model, d_model)` W_Q tensor, then split, but the per-head semantics are identical.
Quick reference
TL;DR
A head owns its own Q, K, V projections of width d_head, runs its own softmax, and produces a d_head-dimensional context vector per token.
Think of a panel of judges scoring an audition. Each judge has a private notebook for what they care about, a private notebook for what each contestant offers, and a private notebook for the contestant's actual material. Every judge scores the contestants independently using only their own three notebooks, never peeking at another judge's notes. After scoring, each judge writes down a short personal verdict. The verdicts are then stapled together and handed to a single producer who reads the whole stack and writes the final result on the show. Each judge is one independent scorer in the layer; the producer at the end is the one who mixes the verdicts back into the show.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Trying to use multi-head attention with a head count that does not divide d_model evenly d_head must be an integer, so n_heads has to divide d_model exactly; otherwise the split is not well-defined and most kernels reject the configuration.
Treating multi-head as a stack of full-width attentions instead of a partition of the model dimension The whole point of MHA is to keep the parameter and compute budget identical to single-head while letting different heads specialize, stacked full-width attentions multiply the cost.
Sharding tensor parallelism at sub-head granularity The softmax is per-head and per-row; splitting inside a head breaks the attention math and requires expensive cross-shard communication.
Quick scan · 2–4 min
Everything important, quickly.
6-8 min: what a head owns + shape of W_Q, W_K, W_V + d_head formula + parallel execution + concat + W_O mixing + GQA and MQA variants.
Real-world examples
Real products, models, and research that use this idea.
1GPT-3 175B: d_model = 12288, n_heads = 96, d_head = 128. Standard multi-head, every head owns its own Q, K, V.
3Llama 4 Maverick and DeepSeek V4 follow the same d_head = 128 convention with GQA-style sharing for KV.
4Gemma 4 uses similar head splits with the modern QK-norm and grouped sharing patterns.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf d_head is fixed at 128, what changes when you scale up d_model?
Sign in or go Pro to see the approach for this follow-up.
QWhy is W_O specifically d_model by d_model and not something else?
Sign in or go Pro to see the approach for this follow-up.
QHow does MQA change what 'a head owns'?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Saying heads share Q, K, and V projections. They do not in standard multi-head attention. Each head has its own (d_model x d_head) projection matrices.