Name the two properties softmax guarantees for every row of the attention weight matrix

Fill in blank·Easy·4.0 · 0·~1 min

Attempt it

Every entry of a softmax-normalized attention row lies in the interval [, ], and the entries of each row sum to exactly .

**Every entry of a softmax-normalized attention row lies in `[0, 1]`, and each row sums to exactly `1`.** The two properties together mean each row is a valid probability distribution over the keys. Mathematically: $$w_{ij} = \frac{e^{s_{ij}}}{\sum_k e^{s_{ik}}}$$ **Why every entry is in `[0, 1]`.** The exponential `e^{s_{ij}}` is strictly positive, so the numerator is positive. The denominator is the sum of all exponentials in the row, which is at least the numerator. So each entry is a positive number divided by a sum that includes itself, which always lies in `(0, 1]`. In practice, after numerical clamping or masking, entries can reach exactly 0 for masked positions but cannot exceed 1. **Why each row sums to 1.** The denominator in every row entry is the same constant: the sum of all exponentials in that row. Adding up the row: $$\sum_j w_{ij} = \sum_j \frac{e^{s_{ij}}}{\sum_k e^{s_{ik}}} = \frac{\sum_j e^{s_{ij}}}{\sum_k e^{s_{ik}}} = 1$$ The numerator and denominator are identical sums, so the row sum is 1 by construction. Together, these two properties make the softmax row a probability distribution over keys, which is what makes the downstream `sum_j w_{ij} v_j` a valid convex combination of value vectors.

**Every entry lies in `[0, 1]` and each row sums to exactly `1`.** These two properties together make each row of the attention weight matrix a valid probability distribution over keys, equivalently a point on the probability simplex. The softmax formula: $$w_{ij} = \frac{e^{s_{ij}}}{\sum_k e^{s_{ik}}}$$ produces both properties by construction. **Non-negativity (`w_{ij} >= 0`).** Every `e^{s_{ij}}` is strictly positive (the exponential of any real number is positive), so both numerator and denominator are positive, so the ratio is positive. Practical implementations can produce exactly 0 for masked positions where `s = -inf`, but never negative values. **Upper bound (`w_{ij} <= 1`).** The denominator includes the numerator as one of its summands, so the ratio cannot exceed 1. Equality `w_{ij} = 1` is approached when one score dominates and all others are negligibly smaller. **Row sum = 1.** Sum the row: $$\sum_j w_{ij} = \sum_j \frac{e^{s_{ij}}}{\sum_k e^{s_{ik}}} = \frac{\sum_j e^{s_{ij}}}{\sum_k e^{s_{ik}}} = 1$$ The numerator and denominator are identical sums. **Why this matters in practice.** Because each row sums to 1, the downstream `sum_j w_{ij} v_j` is a convex combination of value vectors. The output vector lives in the convex hull of the value vectors, with the row entries acting as mixing weights. This probabilistic interpretation is what justifies calling `w_{ij}` an 'attention weight' or an 'attention probability', and it is what makes masking work cleanly: setting `s = -inf` at a masked position drives `w = 0`, the remaining entries automatically renormalize.

Quick reference

TL;DR

Every entry lies in [0, 1] and each row sums to exactly 1. Softmax produces a row-stochastic matrix, a valid probability distribution over keys.

Memory aid

Easy to grasp

Picture cutting a single pizza into slices to share among friends. No friend can get a slice of negative size, no friend can get more than the whole pizza, and the total amount of pizza handed out has to add up to exactly one pizza, no more, no less. The step that turns raw model scores into attention works the same way. Each item on the menu gets a fraction of the model's focus that is at least 0 and at most 1, and the fractions across the whole row always sum to exactly one pizza's worth. That tidy 'no negatives, no overweights, total of one' rule is what makes the row behave like a clean share-out.

Key concepts

Softmax Probability simplex Attention weight Row-stochastic matrix

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

State the two properties (entries in [0, 1], row sums to 1), prove each from the softmax formula, give the probability-simplex interpretation, and walk the downstream consequences (convex-combination output, mask renormalization, attention sinks).

Real-world examples

Real products, models, and research that use this idea.

Vaswani et al. 2017: the canonical attention formula softmax(QK^T / sqrt(d_k)) V produces row-stochastic weight matrices, the [0, 1] entries and row-sum-to-1 are why the output is a convex combination of values.
Attention visualizations (BertViz, exBERT) display the post-softmax weight matrix as a heatmap where each row's cells visually add up to 1.
Causal masking in every modern LLM (GPT-5.5, Llama 4 Maverick, Claude Opus 4.7, Gemini 3.1 Pro) is applied at the score level (set masked positions to -inf), the [0, 1] and sum-to-1 constraints are preserved automatically.
Softmax-1 (Evan Miller 2023, used in some Gemma variants) relaxes the row-sum-to-1 constraint to allow rows summing to less than 1, eliminating the attention-sink consequence.
FlashAttention v2 and v3 maintain running max and sum statistics to compute softmax tile by tile, the [0, 1] and sum-to-1 properties hold for the final output even though the full weight matrix is never materialized.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat changes about the [0, 1] and sum-to-1 properties when softmax-1 is used in place of standard softmax?

QHow does the simplex constraint enable masking to work without explicit renormalization?

QGeometrically, what does the convex-hull interpretation of attention output mean?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Forgetting that weights must be NON-NEGATIVE and at most 1. Softmax cannot produce negative weights or weights larger than 1, regardless of the score values.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.