Given Q of shape (B, n_heads, T, d_head) and the standard self-attention formula Attn(Q, K, V) = softmax(QK^T / sqrt(d_k)) V, the per-head attention output before concatenation has shape .
**Trace the shapes step by step.** Start with Q, K, V each of shape `(B, n_heads, T, d_head)` where B is batch size, T is sequence length, and d_head is the per-head dimension.
The first matmul is `Q @ K^T`. K has shape `(B, n_heads, T, d_head)`; transposing the last two dims gives `(B, n_heads, d_head, T)`. Matmul against Q produces `(B, n_heads, T, T)`. This is the per-head attention-score matrix: for each batch element, for each head, a `(T, T)` matrix saying how strongly each query token attends to each key token.
Scaling by `1 / sqrt(d_head)` and applying softmax along the last dimension does not change shape; the result is still `(B, n_heads, T, T)`. Now every row of the inner matrix is a probability distribution over the T key positions.
The final matmul is `attention_weights @ V`. The weights are `(B, n_heads, T, T)`; V is `(B, n_heads, T, d_head)`. The inner T dimensions collapse, and the result has shape `(B, n_heads, T, d_head)`.
So each query token in each head ends up with a single d_head-wide context vector, which is the weighted sum of all V vectors using the attention weights as the mixing coefficients. The full output tensor has shape `(B, n_heads, T, d_head)`, ready to be concatenated across heads back into `(B, T, d_model)`.
The per-head attention output has shape `(B, n_heads, T, d_head)`, which is identical to the shape of Q (and of K and V).
The arithmetic is short. Start with Q, K, V all of shape `(B, n_heads, T, d_head)`. Compute scores:
$$\text{scores} = \frac{QK^\top}{\sqrt{d_\text{head}}}, \quad \text{shape} = (B, n_\text{heads}, T, T)$$
Apply softmax along the last dimension; shape is unchanged. Multiply by V, where V is `(B, n_heads, T, d_head)`. The last T of the scores collapses with the first T of V, yielding shape `(B, n_heads, T, d_head)`.
The useful intuition: each query token receives one context vector per head. The context vector has dimension d_head because it is a weighted sum of V vectors, and each V vector is d_head-wide.
After this, you typically `transpose` the n_heads axis with the T axis and `view` to merge n_heads with d_head, producing `(B, T, d_model)` where `d_model = n_heads * d_head`. That reshape is the 'concatenate heads' step. The output projection W_O then operates on the d_model dimension.
Memorizing the shape pipeline is foundational: it lets you debug any attention implementation by trace-printing tensor shapes at each step and catching the bug at the first place the shape diverges from the expected sequence.
Quick reference
TL;DR
The per-head output is (B, n_heads, T, d_head). The softmax produces a (T, T) attention matrix per head, then multiplying by V of shape (T, d_head) recovers d_head columns.
Picture a librarian handling T books at once. For each book the librarian asks 'how related are you to every other book in the room?' and gets back a list of T relatedness scores, one per other book. The librarian then makes one summary per book by mixing the contents of every book according to those scores. The summary for each book is a single page exactly as wide as one book's original page count. The librarian does this for every book (T books) and ends up with T summaries, each the same width. Now imagine a whole roomful of librarians doing this in parallel, each focused on a different angle, for several different stories at once. The full pile of summaries has the shape (B, n_heads, T, d_head).
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Cross-attention layers where Q comes from a different sequence than K and V The Q sequence length and K/V sequence length differ; the score matrix is (B, n_heads, T_q, T_kv) and the output shape is (B, n_heads, T_q, d_head).
Sliding-window or sparse attention where each query attends to only a subset of keys The score matrix has a different sparsity pattern, but the final output per query token still has shape (B, n_heads, T, d_head) because each query still produces one context vector.
Linear-attention variants that bypass the (T, T) attention matrix entirely The intermediate shape changes (you compute K^T V as a (d_head, d_head) matrix first), but the final per-head output is still (B, n_heads, T, d_head).
Quick scan · 2–4 min
Everything important, quickly.
Trace shapes from Q, K, V through QK^T, softmax, and the final matmul against V. Distinguish the attention-weight matrix shape (T, T) from the output shape (d_head). Connect to the concat step that recovers d_model, and end with how FlashAttention preserves the output shape while avoiding materializing the weight matrix.
Real products, models, and research that use this idea.
1PyTorch's torch.nn.functional.scaled_dot_product_attention returns output of shape (B, n_heads, T, d_head) by default.
2FlashAttention v2's API takes (B, T, n_heads, d_head) and returns the same shape; internally it tiles over T to avoid materializing the (T, T) attention matrix.
3vLLM's paged attention kernel returns per-head outputs in (B, T, n_heads, d_head) layout for cache-friendly access patterns.
4TransformerLens exposes attention outputs at (B, T, n_heads, d_head) for mechanistic-interpretability analysis.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do the shapes change in cross-attention versus self-attention?
Sign in or go Pro to see the approach for this follow-up.
QWhy does softmax leave the shape unchanged when other reductions change shape?
Sign in or go Pro to see the approach for this follow-up.
QWhat changes in GQA where K and V have fewer heads than Q?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Confusing the shape of the attention-weight matrix (B, n_heads, T, T) with the shape of the attention output (B, n_heads, T, d_head). The matmul against V brings the trailing dimension back to d_head.