Predict what softmax produces when every key in a row is masked out.

The buggy passage makes two wrong claims back to back, and they compound. **First wrong claim:** softmax over an all -inf row returns a uniform distribution. It does not. Softmax is `exp(x_i) / sum_j exp(x_j)`. If every `x_j = -inf`, every `exp(x_j) = 0`, the denominator is the sum of zeros which is `0`, and every output is `0 / 0 = NaN`. There is no uniform fallback baked into the operator. **Second wrong claim:** training continues normally. It does not. A NaN in the attention output flows into the residual stream via the add, then into layer norm (which does a divide), then into the next attention block, then into the loss. Within a handful of forward passes, the entire activation map is NaN, gradients are NaN, the optimizer step pushes parameters to NaN, and the model is wrecked. The fix is to never let a row be fully masked. Either guarantee at least one valid key per query structurally, or detect the all-masked rows and force their attention output to zero (not their softmax row, which is already NaN, but the post-matmul output).

Two compounding errors in the passage. **Softmax of an all -inf row is NaN, not uniform.** The operator is `exp(x_i) / sum_j exp(x_j)`. With every `x_j = -inf`, you get `0 / 0` for every entry. The reason this trips people up is that the limit feels like it should be uniform, but softmax does not take a limit, it evaluates the formula directly, and `0/0` is NaN. Display form for the failure case: $$\text{softmax}(x)_i = \frac{e^{x_i}}{\sum_j e^{x_j}} \quad\Rightarrow\quad \frac{0}{0} = \text{NaN} \text{ when all } x_j = -\infty$$ **Training does not continue normally.** NaN propagates: any arithmetic op with a NaN operand returns NaN. The attention output is NaN, the residual add carries NaN forward, layer norm divides by NaN, the next block reads NaN inputs, the loss is NaN, and the gradients are NaN. A single bad row in one forward pass is enough to corrupt every parameter on the next optimizer step. The defenses are structural or post-hoc. Structurally, design the mask so every query has at least one valid key, for example always include a BOS or sink token in scope. Post-hoc, detect rows where every score is -inf before softmax and force the attention output to zero for those rows. FlashAttention v2 and v3 handle this case explicitly in the kernel; naive PyTorch implementations do not.