Encoder self-attention versus decoder self-attention, what is the single structural difference at the mask level?

Encoder self-attention and decoder self-attention look the same in code except for **one line**: the causal mask. Both use: - The same Q, K, V projections (W_Q, W_K, W_V, each `d_model x d_model`). - The same multi-head reshape. - The same scaled dot-product (`softmax(QK^T / sqrt(d_k))`). - The same output projection W_O. - The same residual connection and layer norm. The difference is what happens to the score matrix before softmax: - **Encoder self-attention:** no mask. Every token can attend to every other token, in both directions. The score matrix is fully populated. - **Decoder self-attention:** causal mask. A lower-triangular mask sets all entries above the diagonal to `-inf` before softmax. Position t can only attend to positions 0 through t, its own past, never the future. Why does the decoder need this? Because it generates one token at a time. At inference, the decoder cannot peek at a future token (it has not been generated yet). At training, the causal mask forces the model to learn this constraint by preventing leakage from later positions. One mask, two architectures.

Encoder and decoder self-attention differ at exactly one place: the causal mask on the score matrix before softmax. **Encoder self-attention** is fully bidirectional. No mask. The score matrix is dense. Token i can attend to all tokens j in 0 to N-1, including future positions. This is the right design for tasks like classification, embedding, and masked-token prediction, where the model should use the full context to understand the input. **Decoder self-attention** is causal. A lower-triangular mask sets `scores[i, j] = -inf` for all `j > i`. Position i can only attend to positions 0 to i. This is required because the decoder generates autoregressively, at inference it cannot see a token it has not yet produced, and training-time causal masking forces the model to learn under this constraint. Everything else in the sub-layer is identical: Q, K, V projection matrices, multi-head reshape, scale by `sqrt(d_k)`, softmax, output projection W_O, residual + norm. The mask is the single structural difference. In code: ```python scores = Q @ K.transpose(-2, -1) / sqrt(d_k) if causal: scores = scores.masked_fill(causal_mask, float('-inf')) A = softmax(scores, dim=-1) out = A @ V ``` One `if` statement, two architectures.