Predict what softmax produces when every key in a row is masked out.
Click any words you think contain an error. Click again to unmark.
An all -inf row gives 0/0 = NaN, not a uniform distribution. The NaN propagates and nukes training; guard fully-masked rows explicitly.
Imagine a voter asked to split 100% of their vote among candidates, but every candidate has been ruled ineligible. The voter cannot pick anyone, but they also cannot leave the form blank, so the form short-circuits and returns garbage. That garbage answer then gets fed into the next person's form, who reads garbage, writes garbage, and passes it on. Within a few hops every form in the building reads garbage. That is exactly what happens when an attention row has every key masked: the math has no valid answer to give, so it returns NaN, and that NaN spreads through every layer that follows.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Softmax over an all -inf row returns NaN, not a uniform distribution. That single fact is the root of a class of bugs that show up in document-aware attention, mixture of experts routing, and any segment-restricted attention pattern. The bug is silent: training appears normal for a few thousand steps, then the loss reports NaN, gradients are NaN, the model is destroyed.
This deep dive walks the algebra, traces NaN propagation through one forward pass, ranks the production defenses, and shows where the failure mode shows up in modern 2026 stacks (document-packed pretraining, MoE routing, StreamingLLM serving).
The right mental model: softmax does not gracefully handle empty support. Treat fully-masked rows as a structural invariant violation that must be prevented, not a runtime case to recover from.
The softmax algebra at all -inf
The canonical softmax formula evaluates each entry as exp(x_i) divided by the sum of all exponentials in the row.
Substituting -inf
When every x_j = -inf, two things happen at once:
- Every numerator
exp(-inf) = 0. - The denominator
sum_j 0 = 0.
Every output entry is 0 / 0, which IEEE-754 defines as NaN. There is no implicit limit, no uniform fallback, no special case in the operator.
Why the numerically-stable form does not save you
Production code uses softmax(x - max(x)) to keep exp arguments non-positive. With every x_j = -inf, max(x) = -inf too. The shift becomes x - max(x) = -inf - (-inf), which IEEE-754 also defines as NaN. The stabilization step itself produces NaN before exp is even called.
Key insight: the stability trick handles overflow, not empty support. Fully-masked rows are a structural problem the operator cannot patch.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- PyTorch documents the fully-masked row trap for nn.functional.scaled_dot_product_attention and recommends user-side guarantees.
- FlashAttention v2 and v3 kernels include explicit handling for rows where every key is masked, returning zero output rather than NaN.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does the numerically-stable softmax form softmax(x - max(x)) also fail on an all -inf row?
max(x) = -inf, so x - max(x) computes -inf - (-inf), which IEEE-754 defines as NaN. Once max is -inf, the stabilization step itself produces NaN before exp is even called. The all -inf case is not patched by the standard stability trick.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming softmax over all -inf gives uniform over masked positions. It gives NaN, which silently breaks training a few thousand steps later.
60 second bullets to scan on the way to the call.
Why softmax of an all -inf row is NaN, with the explicit 0/0 derivation
How NaN propagates through residual, layer norm, and the matmul chain
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.