1Decoder self-attention: Q, K, V all projected from the decoder's own current hidden state, with a causal lower-triangular mask so position t only sees positions <= t.
2Feed-forward sub-layer (FFN): operates per token, no cross-token interaction. Runs on the residual stream and pushes its output back into the residual.
3Cross-attention: Q projected from the decoder hidden state, while K and V are projected from the final-layer encoder output. No causal mask on the encoder axis, the encoder is fully observed.
**The correct order is: causal self-attention, then cross-attention, then position-wise FFN.**
A T5 (and original Transformer 2017) decoder block has three sub-layers, each with its own residual connection and layer norm.
### Sub-layer 1: causal self-attention
The decoder looks at its own previously-generated tokens. Q, K, and V are all projected from the decoder's current hidden state. A causal lower-triangular mask ensures position `t` can only attend to positions `0, 1, ..., t`, not to future positions. This is what makes generation autoregressive.
### Sub-layer 2: cross-attention (the encoder-decoder bridge)
The decoder consults the encoder's output. Q is projected from the decoder's hidden state (the updated state after self-attention). K and V are projected from the encoder's final-layer output, which is fixed across decoder steps. **No causal mask on the encoder axis** because the encoder is fully observed, the decoder can attend to any encoder position.
This is where the encoder and decoder communicate. Cross-attention is what makes encoder-decoder models useful for translation, summarization, and any task with a source-target structure.
### Sub-layer 3: position-wise FFN
A two-layer MLP applied independently to each position. No cross-token interaction. Refines each token's representation based on the combined self-attention and cross-attention output.
### Why this order
The decoder must process its own context first (what has been generated so far) before forming a useful query against the encoder output. Reversing the order, doing cross-attention before self-attention, would force the cross-attention query to be based on the raw input embedding without any awareness of the decoder's own state. The order causal -> cross -> FFN is what enables coherent generation.
**The correct order is: causal self-attention, then cross-attention, then position-wise FFN.**
A T5 decoder block has three sub-layers in this fixed sequence, each wrapped with residual + layer norm.
### Sub-layer 1: decoder self-attention
- Q, K, V all projected from the decoder's current hidden state.
- Causal lower-triangular mask: position `t` attends only to positions `0, 1, ..., t`.
- Updates the decoder's representation based on its own previously-generated tokens.
### Sub-layer 2: cross-attention
- Q projected from the decoder's hidden state (post self attention).
- K and V projected from the **final-layer encoder output**, which is fixed across all decoder steps.
- No causal mask on the encoder axis, the decoder can attend to any encoder position freely.
- This is the encoder-decoder bridge. Without it, the decoder would have no access to the source sequence at all.
### Sub-layer 3: position-wise FFN
- Two-layer MLP applied independently to each position.
- No cross-token interaction.
- Refines each token's representation.
### Why this order matters
The decoder must process its own context (via causal self-attention) before issuing a query against the encoder output (via cross-attention). The cross-attention Q is built from the decoder's UPDATED hidden state, not its raw input. This lets the query be "what do I need from the source given what I have generated so far?" rather than "what does this raw position need from the source?"
The FFN comes last because it has no information to contribute until both attention sub-layers have done their job of moving information between positions and across the encoder-decoder boundary.
**Mnemonic**: review your work (self-attn) -> consult the source (cross-attn) -> polish each word (FFN).
Quick reference
TL;DR
T5 decoder block order: causal self-attention first (Q, K, V from decoder), then cross-attention (Q from decoder, K, V from encoder), then position-wise FFN, each with residual and norm.
Think of a translator writing the next word of a translation. First the translator reviews what they have already written so far (causal self-attention: looking at their own previous work). Then the translator looks at the original sentence to see what still needs translating (cross-attention: consulting the encoder). Finally the translator polishes the wording for the next word individually (FFN: per-token refinement). The order matters because reviewing your own progress before consulting the source means you ask the source the right question.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Decoder-only LLMs for general chat and reasoning (GPT-5.5, Claude Opus 4.7, Llama 4 Maverick) Decoder-only models have no encoder and no cross-attention; the entire input plus output flows through one causal self-attention stack. The encoder-decoder pattern is reserved for tasks with a clean source-target structure.
Encoder-only models for classification or retrieval (BERT, ModernBERT) Encoder-only models do not generate; they pool a single embedding for downstream consumption. No decoder block, no cross-attention, no causal mask.
Very long sequences where caching K, V from the encoder is memory-prohibitive Encoder K, V cache scales linearly with encoder length, for very long source documents (long-context summarization beyond 100k tokens), this can dominate memory; decoder-only or sliding-window encoder-decoder variants may be preferable.
Real-time streaming where the source is also being generated incrementally Encoder-decoder requires the encoder output to be complete before the decoder starts; if the source is itself a streaming signal, a decoder-only or fully autoregressive model is a better fit.
Quick scan · 2–4 min
Everything important, quickly.
6 min: T5 decoder block structure, three sub-layers in causal-self -> cross -> FFN order, projection sources per sub-layer, mask conventions, why causal then cross is the natural ordering, encoder K, V caching, modern survival of encoder-decoder in code, speech, multimodal.
Side-by-side
Sub-layer
Q from
K, V from
Mask
Decoder self-attention
Decoder hidden state
Decoder hidden state
Causal lower-triangular
Cross-attention
Decoder hidden state (post self attn)
Encoder final-layer output
None on encoder axis; optional padding mask
Position-wise FFN
N/A (no attention)
N/A (per-token MLP)
N/A
Real-world examples
Real products, models, and research that use this idea.
1T5 (Raffel et al. 2020) is the canonical text to text encoder-decoder model used as the architectural reference for this layout.
2T5-v1.1 and T5-1.1-XXL retain this exact block structure with GeGLU activations and relative position bias.
3Codestral and other 2026 code-generation models inherit the encoder-decoder pattern for source to target structured tasks.
4Whisper for speech recognition uses encoder-decoder with this block structure: audio encoder feeds K, V into text decoder cross-attention.
5BLIP-2's Q-Former and Flamingo's gated cross-attention extend the cross-attention sub-layer pattern to bridge vision and language, with similar self then cross ordering.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does cross-attention K, V get cached across all decoder steps?
Sign in or go Pro to see the approach for this follow-up.
QHow does the cross-attention pattern in encoder-decoder differ from decoder-only models with system prompts?
Sign in or go Pro to see the approach for this follow-up.
QWhy has the decoder-only paradigm largely replaced encoder-decoder for general LLMs since 2022?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Putting cross-attention before self-attention. The decoder must process its own context first (causal self-attention) before forming a query against the encoder output (cross-attention).