Predict the softmax output for a causal masked attention row.

This is a direct mechanical exercise. You are given the raw attention scores at position 2 of a length-4 sequence as [1.0, 2.0, 3.0, 4.0], and you have to apply causal masking and softmax. Causal masking means position 2 can look at positions 0, 1, and 2, but not at the future position 3. So before computing softmax, you replace position 3's score with negative infinity. The score vector becomes [1.0, 2.0, 3.0, -∞]. Now apply softmax. Softmax exponentiates each score and divides by the sum of all exponentials. Compute the four exponentials: e^1 is about 2.718, e^2 is about 7.389, e^3 is about 20.086, and e^(-∞) is 0. The sum is 30.193. Divide each exponential by the sum to get the weights: 2.718/30.193 is about 0.0900, 7.389/30.193 is about 0.2447, 20.086/30.193 is about 0.6652, and 0/30.193 is 0.0000. The final answer is [0.0900, 0.2447, 0.6652, 0.0000]. The masked position has exactly zero weight, and the three unmasked weights sum to one, a valid probability distribution over only the allowed positions.

Yeah, the way I'd walk through this is: first apply the causal mask, then compute softmax over the masked vector. Position 2 can attend to positions 0 through 2 but not the future position 3. So replace position 3's score with negative infinity: the vector becomes [1, 2, 3, -∞]. The canonical softmax form is: $$\text{softmax}(x)_i = \frac{e^{x_i}}{\sum_j e^{x_j}}$$ Exponentiating gives 2.718, 7.389, 20.086, and 0. The 0 is the key bit, e to the minus infinity collapses exactly to zero, so the masked position contributes nothing to either the numerator or the denominator. The sum is 30.193. Dividing through, the weights are roughly 0.0900, 0.2447, 0.6652, 0.0000. The three unmasked weights still sum to one, which is what you want, a clean probability distribution over only the allowed positions. The trade off worth naming is that the mask has to be applied before softmax, not after. Zeroing a score before softmax leaves the masked position with about 3 percent weight, and zeroing weights after softmax breaks normalization. Negative infinity before softmax is the only clean approach.