Same topic, related formats. Practice these next.
Click any words you think contain an error. Click again to unmark.
An all -inf row gives 0/0 = NaN, not a uniform distribution. The NaN propagates and nukes training; guard fully-masked rows explicitly.
Imagine a voter asked to split 100% of their vote among candidates, but every candidate has been ruled ineligible. The voter cannot pick anyone, but they also cannot leave the form blank, so the form short-circuits and returns garbage. That garbage answer then gets fed into the next person's form, who reads garbage, writes garbage, and passes it on. Within a few hops every form in the building reads garbage. That is exactly what happens when an attention row has every key masked: the math has no valid answer to give, so it returns NaN, and that NaN spreads through every layer that follows.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Derive 0/0 = NaN for the all -inf row, walk NaN propagation through residual and layer norm, rank the three production defenses, and explain how FlashAttention and StreamingLLM avoid the failure mode in practice.
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Assuming softmax over all -inf gives uniform over masked positions. It gives NaN, which silently breaks training a few thousand steps later.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.