For a single attention head with Q of shape (T_query, d_head) and K of shape (T_key, d_head), the product QK^T has shape (T_query, ).
**Q** has shape `(T_query, d_head)`: one row per query token, each row holding the d_head-dimensional query vector. **K** has the same column count but with `T_key` rows: one row per key token.
To compute attention scores between every query and every key, we want a matrix where entry (i, j) is the similarity between query token i and key token j. That is exactly the dot product `q_i . k_j`, which is the matrix product `Q @ K^T`.
The matmul rule: `(a, b) @ (b, c)` gives `(a, c)`. Here Q is `(T_query, d_head)` and K^T is `(d_head, T_key)`. So `QK^T` is `(T_query, T_key)`.
Key reminder: the d_head dimension gets summed over (that is what the dot product does). It does not appear in the output shape. What you see in the output is the two sequence lengths: one row per query, one column per key.
This matrix then gets divided by sqrt(d_head), softmaxed row-wise, and used to weight V to produce the attention output.
QK^T has shape `(T_query, T_key)`. Each entry `(i, j)` is the dot product `q_i . k_j`, the unnormalized similarity between query token i and key token j.
The `d_head` dimension is contracted by the matmul: `(T_query, d_head) @ (d_head, T_key) -> (T_query, T_key)`. The output is a row per query, column per key score grid.
For **self-attention**, `T_query == T_key` because Q, K, V are all projections of the same sequence. The score matrix is square, and any positional mask (causal triangular mask, for instance) is also `(T_query, T_key)`.
For **cross-attention** (encoder-decoder attention in a translation transformer), the two lengths differ: the decoder generates Q with its current generation length, and the encoder supplies K from the source sequence. So if you are translating a 12-word English sentence into French and have generated 5 French tokens, the cross-attention score matrix is `(5, 12)`.
After the scores: divide by sqrt(d_head), softmax along the last (T_key) axis, and matmul with V of shape `(T_key, d_head)` to get the attention output `(T_query, d_head)`.
Quick reference
TL;DR
QK^T has shape (T_query, T_key). Each entry is the dot-product similarity between one query token and one key token.
Imagine a spreadsheet where rows are people asking questions and columns are books that might answer them. Every cell holds a score for how well that book matches that question. If five people ask questions and there are twenty books, you get a five by twenty grid. That grid is QK^T. The row count comes from how many askers there are (queries), and the column count comes from how many books exist (keys). In self-attention the askers and the books are the same set, so the grid is square. In cross-attention they are different sets, so the grid is rectangular.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
When working with linear attention or kernelized attention variants Linear attention rewrites the computation as `phi(Q) (phi(K)^T V)`, never materializing a (T_query, T_key) matrix. The score-matrix shape framing does not directly apply; the bottleneck shifts to d_head x d_head.
When debugging State Space Models (Mamba, RWKV) These do not use attention's QK^T at all; sequence mixing is via a recurrence or selective state update. There is no (T_query, T_key) score tensor.
When reasoning about MoE expert routing scores MoE routing scores have shape (T, n_experts), not (T_query, T_key). Conflating the two is a common confusion when first reading MoE papers.
When computing per-head versus pooled scores in mechanistic interpretability work Some interp analyses average attention patterns across heads or layers, producing aggregated tensors with different shapes. The per-head (T_q, T_k) form is the canonical raw view, not the only useful view.
Quick scan · 2–4 min
Everything important, quickly.
State the shape with the matmul rule, explain self vs cross-attention shape symmetry, connect to O(n^2) memory and FlashAttention, mention causal mask shares the same shape.
Real-world examples
Real products, models, and research that use this idea.
1Llama 4 with H_q=64 heads at 128k context has per-layer score tensor of shape (B, 64, 131072, 131072), which is why FlashAttention is mandatory.
2Encoder-decoder translation in NLLB-200 has cross-attention score matrices of shape (target_seq_len, source_seq_len) per layer.
3Stable Diffusion 3 / SD3.5 cross-attention between image latents and CLIP text embeddings produces (image_tokens, text_tokens) score matrices.
4Mistral's sliding-window attention restricts the score matrix to a banded structure of width 4096, never materializing the full (T, T) tensor.
5Whisper's encoder-decoder cross-attention has scores of shape (audio_frames, decoder_step) per layer.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf the score matrix is (T_query, T_key), what is the shape of the attention output, and how do you get there?
Sign in or go Pro to see the approach for this follow-up.
QWalk through how FlashAttention computes the output without materializing the full (T_query, T_key) matrix.
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Writing d_head as the second dimension. The d_head axis gets contracted by the matmul, what remains is the two sequence lengths.