Every entry of a softmax-normalized attention row lies in the interval [, ], and the entries of each row sum to exactly .
**Every entry of a softmax-normalized attention row lies in `[0, 1]`, and each row sums to exactly `1`.**
The two properties together mean each row is a valid probability distribution over the keys. Mathematically:
$$w_{ij} = \frac{e^{s_{ij}}}{\sum_k e^{s_{ik}}}$$
**Why every entry is in `[0, 1]`.** The exponential `e^{s_{ij}}` is strictly positive, so the numerator is positive. The denominator is the sum of all exponentials in the row, which is at least the numerator. So each entry is a positive number divided by a sum that includes itself, which always lies in `(0, 1]`. In practice, after numerical clamping or masking, entries can reach exactly 0 for masked positions but cannot exceed 1.
**Why each row sums to 1.** The denominator in every row entry is the same constant: the sum of all exponentials in that row. Adding up the row:
$$\sum_j w_{ij} = \sum_j \frac{e^{s_{ij}}}{\sum_k e^{s_{ik}}} = \frac{\sum_j e^{s_{ij}}}{\sum_k e^{s_{ik}}} = 1$$
The numerator and denominator are identical sums, so the row sum is 1 by construction.
Together, these two properties make the softmax row a probability distribution over keys, which is what makes the downstream `sum_j w_{ij} v_j` a valid convex combination of value vectors.
**Every entry lies in `[0, 1]` and each row sums to exactly `1`.** These two properties together make each row of the attention weight matrix a valid probability distribution over keys, equivalently a point on the probability simplex.
The softmax formula:
$$w_{ij} = \frac{e^{s_{ij}}}{\sum_k e^{s_{ik}}}$$
produces both properties by construction.
**Non-negativity (`w_{ij} >= 0`).** Every `e^{s_{ij}}` is strictly positive (the exponential of any real number is positive), so both numerator and denominator are positive, so the ratio is positive. Practical implementations can produce exactly 0 for masked positions where `s = -inf`, but never negative values.
**Upper bound (`w_{ij} <= 1`).** The denominator includes the numerator as one of its summands, so the ratio cannot exceed 1. Equality `w_{ij} = 1` is approached when one score dominates and all others are negligibly smaller.
**Row sum = 1.** Sum the row:
$$\sum_j w_{ij} = \sum_j \frac{e^{s_{ij}}}{\sum_k e^{s_{ik}}} = \frac{\sum_j e^{s_{ij}}}{\sum_k e^{s_{ik}}} = 1$$
The numerator and denominator are identical sums.
**Why this matters in practice.** Because each row sums to 1, the downstream `sum_j w_{ij} v_j` is a convex combination of value vectors. The output vector lives in the convex hull of the value vectors, with the row entries acting as mixing weights. This probabilistic interpretation is what justifies calling `w_{ij}` an 'attention weight' or an 'attention probability', and it is what makes masking work cleanly: setting `s = -inf` at a masked position drives `w = 0`, the remaining entries automatically renormalize.
Quick reference
TL;DR
Every entry lies in [0, 1] and each row sums to exactly 1. Softmax produces a row-stochastic matrix, a valid probability distribution over keys.
Picture cutting a single pizza into slices to share among friends. No friend can get a slice of negative size, no friend can get more than the whole pizza, and the total amount of pizza handed out has to add up to exactly one pizza, no more, no less. The step that turns raw model scores into attention works the same way. Each item on the menu gets a fraction of the model's focus that is at least 0 and at most 1, and the fractions across the whole row always sum to exactly one pizza's worth. That tidy 'no negatives, no overweights, total of one' rule is what makes the row behave like a clean share-out.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Models trained with softmax-1, which allows row sums to be less than 1 Adding a +1 to the softmax denominator relaxes the row-sum constraint, so the [0, 1] entries are still valid but the row sum is no longer pinned to 1.
Linear attention or kernel attention without explicit softmax Performer-style linear attention factors softmax through a feature map, and the row-sum-to-1 constraint applies differently after factorization; the simplex interpretation may not hold cleanly.
Sparsemax-based attention Sparsemax produces a row with exact zeros for low-score positions; the row still sums to 1 but lives on the simplex boundary, so the entry range and sum constraint are preserved but the support is sparse.
Hard attention with argmax in place of softmax Argmax produces a one-hot row (one entry = 1, rest = 0); the simplex constraint is preserved but the row is at a vertex of the simplex rather than the interior.
Quick scan · 2–4 min
Everything important, quickly.
State the two properties (entries in [0, 1], row sums to 1), prove each from the softmax formula, give the probability-simplex interpretation, and walk the downstream consequences (convex-combination output, mask renormalization, attention sinks).
Real-world examples
Real products, models, and research that use this idea.
1Vaswani et al. 2017: the canonical attention formula softmax(QK^T / sqrt(d_k)) V produces row-stochastic weight matrices, the [0, 1] entries and row-sum-to-1 are why the output is a convex combination of values.
2Attention visualizations (BertViz, exBERT) display the post-softmax weight matrix as a heatmap where each row's cells visually add up to 1.
3Causal masking in every modern LLM (GPT-5.5, Llama 4 Maverick, Claude Opus 4.7, Gemini 3.1 Pro) is applied at the score level (set masked positions to -inf), the [0, 1] and sum-to-1 constraints are preserved automatically.
4Softmax-1 (Evan Miller 2023, used in some Gemma variants) relaxes the row-sum-to-1 constraint to allow rows summing to less than 1, eliminating the attention-sink consequence.
5FlashAttention v2 and v3 maintain running max and sum statistics to compute softmax tile by tile, the [0, 1] and sum-to-1 properties hold for the final output even though the full weight matrix is never materialized.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat changes about the [0, 1] and sum-to-1 properties when softmax-1 is used in place of standard softmax?
Sign in or go Pro to see the approach for this follow-up.
QHow does the simplex constraint enable masking to work without explicit renormalization?
Sign in or go Pro to see the approach for this follow-up.
QGeometrically, what does the convex-hull interpretation of attention output mean?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Forgetting that weights must be NON-NEGATIVE and at most 1. Softmax cannot produce negative weights or weights larger than 1, regardless of the score values.