Position 2 (0-indexed) in a length-4 sequence has raw attention scores [1.0, 2.0, 3.0, 4.0] before masking. Apply causal masking (position 2 can attend to positions 0, 1, 2 but NOT position 3), then softmax. What are the 4 output weights, in order?
This is a direct mechanical exercise. You are given the raw attention scores at position 2 of a length-4 sequence as [1.0, 2.0, 3.0, 4.0], and you have to apply causal masking and softmax.
Causal masking means position 2 can look at positions 0, 1, and 2, but not at the future position 3. So before computing softmax, you replace position 3's score with negative infinity. The score vector becomes [1.0, 2.0, 3.0, -∞].
Now apply softmax. Softmax exponentiates each score and divides by the sum of all exponentials. Compute the four exponentials: e^1 is about 2.718, e^2 is about 7.389, e^3 is about 20.086, and e^(-∞) is 0. The sum is 30.193.
Divide each exponential by the sum to get the weights: 2.718/30.193 is about 0.0900, 7.389/30.193 is about 0.2447, 20.086/30.193 is about 0.6652, and 0/30.193 is 0.0000.
The final answer is [0.0900, 0.2447, 0.6652, 0.0000]. The masked position has exactly zero weight, and the three unmasked weights sum to one, a valid probability distribution over only the allowed positions.
Yeah, the way I'd walk through this is: first apply the causal mask, then compute softmax over the masked vector.
Position 2 can attend to positions 0 through 2 but not the future position 3. So replace position 3's score with negative infinity: the vector becomes [1, 2, 3, -∞].
The canonical softmax form is:
$$\text{softmax}(x)_i = \frac{e^{x_i}}{\sum_j e^{x_j}}$$
Exponentiating gives 2.718, 7.389, 20.086, and 0. The 0 is the key bit, e to the minus infinity collapses exactly to zero, so the masked position contributes nothing to either the numerator or the denominator. The sum is 30.193.
Dividing through, the weights are roughly 0.0900, 0.2447, 0.6652, 0.0000. The three unmasked weights still sum to one, which is what you want, a clean probability distribution over only the allowed positions.
The trade off worth naming is that the mask has to be applied before softmax, not after. Zeroing a score before softmax leaves the masked position with about 3 percent weight, and zeroing weights after softmax breaks normalization. Negative infinity before softmax is the only clean approach.
Quick reference
TL;DR
Mask sets position 3's score to negative infinity before softmax, giving the four weights approximately 0.0900, 0.2447, 0.6652, 0.0000.
Imagine a teacher converting test scores into slices of a pie chart where every slice has to add up to one whole pie. To remove a student entirely, you can't just give them a score of zero, a zero score still claims a sliver of the pie, because the conversion rule turns zero into a real positive number first. You have to give them an impossibly bad score, think 'negative infinity bad', which then converts to no pie at all. The remaining students share the whole pie among themselves, like causal masking sweeps a future position out of the attention distribution.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Bidirectional encoders like BERT during pretraining or fine-tuning Bidirectional models deliberately let every position attend everywhere; applying a causal mask removes the property the architecture was designed for.
Cross-attention from decoder to encoder over a fully-visible source sequence The source side is not autoregressive; masking out source positions would cripple the decoder's ability to ground its output in the source.
Bag of tokens or set-style architectures with no ordering Causal ordering is meaningless without a sequence; a causal mask would impose an arbitrary order and degrade quality.
Padding-only masking on the source side without causal structure Padding masks zero out invalid token slots, not future positions, and have different shape and semantics than a causal mask.
Quick scan · 2–4 min
Everything important, quickly.
5 min: masking step, exponentials, sum, divide; the negative-infinity vs zero distinction; max subtraction softmax stabilization; dtype-aware mask values; FlashAttention's block level masking; the all masked edge case.
Real products, models, and research that use this idea.
1PyTorch's torch.nn.functional.scaled_dot_product_attention with is_causal=True applies the canonical mask internally and selects an efficient backend.
2HuggingFace transformers uses torch.finfo(dtype).min as the mask value, which is roughly -3.4e38 for fp32.
3FlashAttention 2 and 3 skip fully masked blocks entirely and apply the boundary mask inside partial blocks to preserve the streaming accumulator's correctness.
4Karpathy's nanoGPT uses tril plus masked_fill with float('-inf') as the reference implementation pattern.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy use a large finite negative instead of literal negative infinity in production kernels?
Sign in or go Pro to see the approach for this follow-up.
QHow does the max subtraction stabilization handle a masked score?
Sign in or go Pro to see the approach for this follow-up.
QWhat if every position in a row is masked?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Zero as the mask value leaks ~3% weight to the masked slot. e^0 is 1, not 0. Only negative infinity vanishes.