Encoder self-attention is bidirectional (no mask); decoder self-attention is causal (lower-triangular mask). Everything else in the sub-layer is identical.
Picture a room of 10 people each writing a story together. In the encoder room, every person can read every other person's notes before writing their own line, full context. In the decoder room, each person can only read notes from people who finished earlier, no peeking at what comes next. The room layouts are identical, the writers use the same pens and the same paper, only one rule changes: who can read whom. That rule is the causal mask.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The encoder versus decoder distinction is one of the most over-explained topics in transformer literature. Diagrams suggest two completely different sub-layers, with separate boxes for 'multi-head attention' versus 'masked multi-head attention'. The reality is one mask. Everything else is identical. Internalizing this is the difference between thinking the transformer architecture is a zoo of different attention types and seeing it as one mechanism with three masking variations.
This deep dive walks the exact code-level difference, justifies the causal mask from autoregressive training and inference, surveys how modern kernels exploit the mask for speedup, and ends with the variants that compose causal masking with other constraints.
What is actually different
Strip away the textbook diagrams and look at the code. Encoder self-attention and decoder self-attention differ at exactly one place.
The shared code path
Both start identically:
Q = x @ W_Q # (B, T, d_model)
K = x @ W_K
V = x @ W_V
Q = Q.view(B, T, H, D).transpose(1, 2) # (B, H, T, D)
K = K.view(B, T, H, D).transpose(1, 2)
V = V.view(B, T, H, D).transpose(1, 2)
scores = Q @ K.transpose(-2, -1) / sqrt(D) # (B, H, T, T)
The one different line
Decoder:
causal_mask = torch.triu(torch.ones(T, T, dtype=torch.bool), diagonal=1)
scores = scores.masked_fill(causal_mask, float('-inf'))
Encoder: nothing. The score matrix is left dense.
After the mask
Both continue identically:
A = softmax(scores, dim=-1)
out = A @ V
out = out.transpose(1, 2).contiguous().view(B, T, d_model)
out = out @ W_O
Why -inf and not 0
Setting masked entries to 0 would let exp(0) = 1 survive softmax, diluting legitimate weights. Setting to -inf gives exp(-inf) = 0, the mathematically correct zero weight. PyTorch's masked_fill with float('-inf') is the canonical idiom.
One mask. One line. Two architectures.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Component | Encoder self-attention | Decoder self-attention |
|---|---|---|
| W_Q, W_K, W_V projections | d_model x d_model | d_model x d_model |
| Multi-head reshape | Standard split into num_heads | Standard split into num_heads |
| Scale | 1 / sqrt(d_k) | 1 / sqrt(d_k) |
| Causal mask | None | Lower-triangular, scores above diagonal = -inf |
| Softmax + V matmul | Standard | Standard |
| Output projection W_O | d_model x d_model | d_model x d_model |
Real products, models, and research that use this idea.
- BERT (encoder-only): every encoder block uses no causal mask; the bidirectional context is the whole point of the architecture.
- Llama 4 Maverick (decoder-only): every block applies a causal mask to the score matrix; FlashAttention-3 skips the upper-triangular blocks for a 2x kernel speedup.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would training break if you forgot to apply the causal mask on the decoder?
The model would see the target token when predicting it because the attention would let position t read from positions > t. Training loss would drop to near-zero immediately as the model learned to copy from the future. At inference, the model would produce nonsense because the future tokens it relied on at training time do not exist at generation time.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Listing many architectural differences between encoder and decoder self-attention. The Q, K, V projections, multi-head structure, and output projection are identical; only the mask differs.
60 second bullets to scan on the way to the call.
Single structural difference: the causal mask
Mask shape: lower triangular, blocks scores above the diagonal
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.