How does softmax turn an attention score into an attention weight?
Score is the pre-softmax raw dot product (any real number). Weight is the post-softmax probability in [0, 1] that rows sum to 1. Softmax is the bridge.
Think of judges at a talent show. First, each judge writes a raw score for every act on a piece of paper. The numbers can be anything: positive, negative, all over the place, and there is no rule that they have to add up to anything. Then the host runs through the room collecting those scores, runs them through a calculator, and converts them so each act lands as a clean percentage between 0% and 100%, with all the percentages across acts adding up to exactly 100%. The raw paper numbers are what the model calls 'scores'. The final percentages are what it calls 'weights'. The host with the calculator in between is the single step that converts one into the other.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Attention score and attention weight refer to different stages of the attention computation, separated by exactly one operation: softmax. They are not synonyms, and conflating them is a sign you have not internalized where in the attention pipeline each modification lives.
This deep dive defines both objects rigorously, walks why every important attention modification (masking, temperature, position biases, stable softmax, FlashAttention) operates at the score level, and explains why the score versus weight distinction is load-bearing for senior-level systems work.
Mental model: scores are raw similarities, weights are probabilities. Softmax is the only operator that crosses the boundary.
Definitions and domains
Attention score
For a query position i and a key position j, the attention score is:
s_{ij} = (q_i · k_j) / sqrt(d_k)
- Domain: all real numbers.
- Row sum: unconstrained.
- Shape: the score matrix is (T_q, T_k) per head.
- Interpretation: raw similarity between query i and key j, with the sqrt(d_k) division already applied to keep variance manageable.
Attention weight
Applying softmax row-wise produces the attention weight:
- Domain:
[0, 1]. - Row sum: exactly 1 for every query i.
- Shape: same as the score matrix, (T_q, T_k) per head.
- Interpretation: the probability that query i attends to key j, equivalently the convex combination weight applied to value vector j when forming the output for query i.
The output formula
The attention output for query i is o_i = sum_j w_{ij} v_j, a convex combination of value vectors. The weight matrix is what enters this combination, not the score matrix.
Key insight: the score matrix is shapeless probability mass. The weight matrix is a row-stochastic matrix where each row lives on the probability simplex.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Vaswani et al. 2017: the canonical attention diagram explicitly separates 'MatMul + Scale' (score) from 'SoftMax' (weight) as distinct boxes.
- Attention visualization tools (BertViz, exBERT) plot the weight matrix, not the score matrix; that is why the heatmap rows sum to 1.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy are causal masks applied at the score level rather than the weight level?
Adding -inf to scores at j > i produces exp(-inf) = 0 in the softmax numerator, so those positions automatically get zero weight while the remaining row entries renormalize correctly via the standard softmax. Doing it post-softmax would require explicit renormalization (zero the masked entries, then divide each remaining entry by the new row sum) to preserve the probability constraint. Score-level masking is mathematically cleaner and computationally cheaper.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Using 'score' and 'weight' interchangeably. They are different objects living in different spaces; softmax is the operator that converts one to the other.
60 second bullets to scan on the way to the call.
Definition of attention score vs attention weight
What domain each one lives in (R vs [0, 1])
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.