**The correct answer is B: along the key axis (the last dimension), so each query gets a probability distribution over keys.**
The score matrix `s = QK^T / sqrt(d_k)` has shape `(T_query, T_key)` per head. Softmax is applied row-wise, where each row corresponds to one query token and the entries within that row correspond to all the keys that query can attend to.
After softmax, each row is a probability distribution: every entry lies in `[0, 1]`, and the entries sum to exactly 1. For each query token, the model has decided how to distribute its attention mass across all the keys.
In PyTorch the call is `softmax(scores, dim=-1)` because the key axis is the last dimension of the score tensor.
**Why the other options are wrong.**
- **Option A (query axis):** Would give each key a probability distribution over queries, which is backwards. The downstream computation `attention_weights @ V` requires each query to have a weighting over keys (and therefore over rows of V), not each key to have a weighting over queries.
- **Option C (batch axis):** Softmax over batch is nonsensical because batch elements are independent samples; you do not want one batch element's scores to depend on another's.
- **Option D (head axis):** Heads are independent attention computations; softmax over heads would couple them and destroy the multi-head design.
**The downstream consequence.** The attention output is `o = softmax(s, dim=-1) @ V`, shape `(T_query, d_v)`. For each query, the output is a convex combination of value vectors with the row's attention weights as mixing coefficients. This only works because each row sums to 1.
**Along the key axis (the last dimension of the score matrix).** Each query gets its own probability distribution over keys, and each row of the post-softmax weight matrix sums to exactly 1.
The score matrix per head has shape `(T_query, T_key)`. The softmax operator is applied row-wise:
$$w_{ij} = \frac{e^{s_{ij}}}{\sum_k e^{s_{ik}}}$$
where the sum in the denominator runs over the key axis. In PyTorch this is `softmax(scores, dim=-1)`.
**Why row-wise specifically.** The downstream computation is:
`output = attention_weights @ V`
where `V` has shape `(T_key, d_v)` and the output is `(T_query, d_v)`. For each query, the output is a convex combination of value rows. That convex combination requires the weights to sum to 1 over the key axis, which is exactly what row-wise softmax produces.
**Why not the other axes.**
- **Query axis (option A):** would give each key a distribution over queries, which is the wrong tensor shape for the downstream matmul. The output would be `T_key`-dimensional in a meaningless way.
- **Batch axis (option C):** batch elements are independent samples in a minibatch; softmax across batch would couple them, which is never what you want.
- **Head axis (option D):** heads are designed to be independent retrieval channels; softmax across heads would destroy that independence.
**The systems perspective.** Score-level operations (masking via -inf, temperature scaling, ALiBi bias) all operate BEFORE softmax and affect the same key axis. Causal masking sets `s_{ij} = -inf` for `j > i` along the key axis, so each row's mask is applied along the key dimension and softmax automatically renormalizes the unmasked entries. All of this fits the 'each query gets a distribution over keys' picture; flipping the axis would break the whole pipeline.
Quick reference
TL;DR
Along the key axis (the last dimension). Each query gets a probability distribution over keys; each row sums to 1.
Imagine planning a meal where each chef has to decide how much of each ingredient to use. The chef does not normalize 'how much each ingredient gets used across all chefs'. The chef normalizes 'how much of my own meal each ingredient takes up', so for each chef's own dish, the ingredient percentages add up to 100%. Each query token in attention is like one chef: it decides for itself how attention mass is split across all available keys, and those splits add up to 100% for that query.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Bidirectional attention models like BERT where the score matrix is symmetric in spirit Even in bidirectional attention, softmax is still applied along the key axis per query; the symmetry is in what each query can see, not in how the normalization axis is chosen.
Linear attention or kernel attention without explicit softmax Performer-style attention factors softmax through a feature map; the explicit row-wise normalization is no longer a single operator and the per-query weighting is implicit.
Models using softmax-1 or other row sum to less than 1 variants The axis remains the key axis, but the row no longer sums to exactly 1; the convex-combination interpretation is weakened.
Cross-attention where K and V come from a different sequence than Q Same axis choice (softmax along the key axis, which is now the encoder sequence's positions), but T_key differs from T_query. The mechanism is identical; only the semantic interpretation differs.
Quick scan · 2–4 min
Everything important, quickly.
Identify the key axis as the normalization axis, derive the choice from the downstream V matmul shape, contrast with wrong-axis options, and walk how masking, FlashAttention tiling, and cross-attention all align with the key-axis convention.
Real-world examples
Real products, models, and research that use this idea.
1PyTorch's torch.nn.functional.scaled_dot_product_attention computes softmax(scores, dim=-1) internally before multiplying by V.
2All modern LLMs (GPT-5.5, Llama 4 Maverick, Claude Opus 4.7, Gemini 3.1 Pro, DeepSeek V4) apply softmax along the key axis in every attention layer.
3Causal masking in autoregressive transformers sets the upper triangular of the score matrix to -inf along the key axis before softmax.
4FlashAttention's online-softmax tiling iterates over key-axis blocks precisely because softmax normalization happens along the key axis.
5Attention visualization tools (BertViz, exBERT) plot the weight matrix with rows summing to 1, confirming the key-axis normalization.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat would change if softmax were applied along the query axis instead of the key axis?
Sign in or go Pro to see the approach for this follow-up.
QHow does FlashAttention's online-softmax algorithm exploit the key-axis normalization?
Sign in or go Pro to see the approach for this follow-up.
QHow does cross-attention preserve the key-axis softmax even though T_query and T_key differ?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Applying softmax along the query axis. That would give each key a distribution over queries, not each query a distribution over keys, and the downstream V matmul would be malformed.