Encoder self-attention and decoder self-attention look the same in code except for **one line**: the causal mask.
Both use:
- The same Q, K, V projections (W_Q, W_K, W_V, each `d_model x d_model`).
- The same multi-head reshape.
- The same scaled dot-product (`softmax(QK^T / sqrt(d_k))`).
- The same output projection W_O.
- The same residual connection and layer norm.
The difference is what happens to the score matrix before softmax:
- **Encoder self-attention:** no mask. Every token can attend to every other token, in both directions. The score matrix is fully populated.
- **Decoder self-attention:** causal mask. A lower-triangular mask sets all entries above the diagonal to `-inf` before softmax. Position t can only attend to positions 0 through t, its own past, never the future.
Why does the decoder need this? Because it generates one token at a time. At inference, the decoder cannot peek at a future token (it has not been generated yet). At training, the causal mask forces the model to learn this constraint by preventing leakage from later positions.
One mask, two architectures.
Encoder and decoder self-attention differ at exactly one place: the causal mask on the score matrix before softmax.
**Encoder self-attention** is fully bidirectional. No mask. The score matrix is dense. Token i can attend to all tokens j in 0 to N-1, including future positions. This is the right design for tasks like classification, embedding, and masked-token prediction, where the model should use the full context to understand the input.
**Decoder self-attention** is causal. A lower-triangular mask sets `scores[i, j] = -inf` for all `j > i`. Position i can only attend to positions 0 to i. This is required because the decoder generates autoregressively, at inference it cannot see a token it has not yet produced, and training-time causal masking forces the model to learn under this constraint.
Everything else in the sub-layer is identical: Q, K, V projection matrices, multi-head reshape, scale by `sqrt(d_k)`, softmax, output projection W_O, residual + norm. The mask is the single structural difference.
In code:
```python
scores = Q @ K.transpose(-2, -1) / sqrt(d_k)
if causal:
scores = scores.masked_fill(causal_mask, float('-inf'))
A = softmax(scores, dim=-1)
out = A @ V
```
One `if` statement, two architectures.
Quick reference
TL;DR
Encoder self-attention is bidirectional (no mask); decoder self-attention is causal (lower-triangular mask). Everything else in the sub-layer is identical.
Picture a room of 10 people each writing a story together. In the encoder room, every person can read every other person's notes before writing their own line, full context. In the decoder room, each person can only read notes from people who finished earlier, no peeking at what comes next. The room layouts are identical, the writers use the same pens and the same paper, only one rule changes: who can read whom. That rule is the causal mask.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Pure classification or embedding tasks with bidirectional context Encoder self-attention is correct here; applying a causal mask would block legitimate forward-reading and degrade representation quality.
Sequence to sequence tasks where the source is fully known in advance The encoder should be bidirectional; only the decoder's self-attention needs causal masking.
Models that need to attend across full document boundaries during pretraining Document-aware causal masking is a third variant that prevents attention crossing document boundaries within a packed training sequence; pure causal masking is wrong here.
Quick scan · 2–4 min
Everything important, quickly.
State the one structural difference (the causal mask), describe its shape and placement (lower-triangular, applied before softmax on the score matrix), justify it from autoregressive generation, and close with how FlashAttention exploits it for kernel speedup.
Side-by-side
Component
Encoder self-attention
Decoder self-attention
W_Q, W_K, W_V projections
d_model x d_model
d_model x d_model
Multi-head reshape
Standard split into num_heads
Standard split into num_heads
Scale
1 / sqrt(d_k)
1 / sqrt(d_k)
Causal mask
None
Lower-triangular, scores above diagonal = -inf
Softmax + V matmul
Standard
Standard
Output projection W_O
d_model x d_model
d_model x d_model
Real-world examples
Real products, models, and research that use this idea.
1BERT (encoder-only): every encoder block uses no causal mask; the bidirectional context is the whole point of the architecture.
2Llama 4 Maverick (decoder-only): every block applies a causal mask to the score matrix; FlashAttention-3 skips the upper-triangular blocks for a 2x kernel speedup.
3T5 (encoder-decoder): the encoder stack uses no mask; the decoder stack applies a causal mask to its self-attention sub-layer.
4Mistral 7B: causal mask plus a sliding window, position i attends only to positions [i - 4096, i] within the sequence.
5PyTorch's `F.scaled_dot_product_attention(..., is_causal=True)`: the canonical API for declaring the causal mask at the kernel level.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would training break if you forgot to apply the causal mask on the decoder?
Sign in or go Pro to see the approach for this follow-up.
QWhy is causal attention faster than bidirectional in modern kernels even though both compute roughly the same matmul shape?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Listing many architectural differences between encoder and decoder self-attention. The Q, K, V projections, multi-head structure, and output projection are identical; only the mask differs.