How does softmax turn an attention score into an attention weight?

The two terms look similar but refer to different stages of the attention computation. **Attention score** is the raw pre-softmax value. For a query position i attending to a key position j, the score is `(q_i · k_j) / sqrt(d_k)`. Scores are unbounded: they can be any real number, often negative, and a row of the score matrix does not sum to anything in particular. **Attention weight** is the post-softmax value. After applying softmax row-wise: $$w_{ij} = \frac{e^{s_{ij}}}{\sum_k e^{s_{ik}}}$$ Each weight is in `[0, 1]`, and the weights in any row sum to exactly 1, so each row is a valid probability distribution over keys. Softmax is the single operator that crosses the boundary. Before softmax: unbounded real values, the 'score' matrix. After softmax: probabilities on a simplex, the 'weight' matrix. The weight matrix is what multiplies V to produce the attention output. The terminology matters because diagrams and papers use both words, and confusing them is a sign you have not internalized the softmax step.

**Score = pre-softmax. Weight = post-softmax. Softmax is the bridge.** The attention score is the scaled dot product `(q · k) / sqrt(d_k)`. Scores are unbounded reals: they can be positive, negative, large, or small, and the row sum is not constrained. They encode raw similarity between a query and a key, with the sqrt(d_k) division already applied so the variance stays manageable. The attention weight is what softmax produces from those scores: $$w_{ij} = \frac{e^{s_{ij}}}{\sum_k e^{s_{ik}}}$$ Each weight lies in `[0, 1]`, and `sum_j w_{ij} = 1` for every query i. The post-softmax row is a probability distribution over keys, living on the probability simplex. **Why the distinction matters in practice.** When papers say 'attention map', they almost always mean weights, the visualizable heatmap that sums to 1 per row. When they say 'score' or 'logit', they mean the pre-softmax matrix. Masking is applied at the score level (set to -inf before softmax). Temperature is also applied at the score level (divide by T before softmax). Relative position biases like ALiBi are added at the score level. The weight matrix is the downstream consumer of all those score-time modifications, and it is what multiplies V to produce the attention output.