Order the sublayers a token visits inside one encoder-decoder decoder block
- 1Decoder hidden states from the previous block arrive as input
- 2The result passes on to the next decoder block
- 3Cross-attention: queries from the decoder, keys and values from the encoder output, plus residual add
- 4Masked causal self-attention over decoder positions, plus residual add
- 5Position wise feed-forward network, plus residual add
Encoder-decoder decoder block: masked self-attention, then cross-attention to the encoder, then FFN, each with a residual add.
Imagine you are writing the next sentence of a translation. First you re-read what you have already written so you do not contradict yourself; that is masked self-attention over your own past words. Then you glance back at the original foreign sentence to see what idea comes next; that is cross-attention, where your half-written translation asks questions and the source sentence supplies the answers. Finally you do a little private thinking to mix and refine what you just heard; that is the feed-forward network. After each of those three steps you also keep what you had before by adding the old version on top, so nothing gets erased. Then the next decoder block does the same three steps again on the cleaner draft.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
An encoder-decoder decoder block is the workhorse of sequence-to-sequence models like T5, BART, and Whisper. Inside one block, a token visits three sublayers in a strict order, each followed by a residual add. The order is not arbitrary: it encodes how the decoder reasons. First it organises its own state. Then it consults the source. Then it refines.
The goal of this walkthrough is to make the order feel inevitable rather than memorised. Once you can answer 'what changes between sublayer one and sublayer two,' you will never confuse this block with a decoder-only block again.
We will work through each sublayer, see why it sits where it sits, and end with how the same recipe shows up in modern translation, summarization, and speech-to-text systems.
Sublayer 1: masked causal self-attention
The first sublayer is masked causal self-attention over the decoder's own positions. The decoder's residual stream enters; the sublayer projects it three ways to form Q, K, and V; the attention is computed with a triangular mask that hides future tokens.
This is the same self-attention you see in GPT-2 or Llama. The point of running it first is that it is the only sublayer that lets the decoder integrate information across its own past positions. Without it, the cross-attention later would have nothing position-specific to ask the encoder about; every decoder position would emit the same query distribution against the same encoder K/V and you would lose the autoregressive distinction between step t and step t plus one.
The attention output is added back into the residual stream via the skip connection. The residual is what makes a 24-layer or 48-layer decoder trainable; pull it out and depth becomes catastrophically unstable. Modern stacks place a pre-norm (RMSNorm in T5 v1.1 and beyond) on the input to this sublayer rather than after the residual add, which improves gradient flow through depth.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# Encoder-decoder decoder block (pre-norm), single layer
class DecoderBlock(nn.Module):
def __init__(self, d_model, n_heads, d_ff):
super().__init__()
self.norm1 = nn.RMSNorm(d_model)
self.self_attn = MaskedSelfAttention(d_model, n_heads)
self.norm2 = nn.RMSNorm(d_model)
self.cross_attn = CrossAttention(d_model, n_heads)
self.norm3 = nn.RMSNorm(d_model)
self.ffn = FFN(d_model, d_ff)
def forward(self, x, enc_kv, causal_mask):
x = x + self.self_attn(self.norm1(x), causal_mask) # sublayer 1
x = x + self.cross_attn(self.norm2(x), enc_kv) # sublayer 2
x = x + self.ffn(self.norm3(x)) # sublayer 3
return xReal products, models, and research that use this idea.
- T5 and its instruction-tuned descendants (Flan-T5) use this exact decoder-block recipe with pre-norm RMSNorm.
- The original 2017 Vaswani transformer used post-norm encoder-decoder with this sublayer order; the order survived; the norm placement migrated to pre-norm.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is encoder K/V computed once but decoder K/V recomputed per step in cross-attention serving?
The encoder reads a fixed source sequence whose hidden states do not change during decoding, so K and V from the encoder side are constants. The decoder grows token by token, so its own cached K and V need an append each step. Cross-attention pulls the static encoder K/V and applies fresh decoder queries.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Swapping the order of self-attention and cross-attention, or forgetting that an encoder-decoder decoder has three sublayers per block while a decoder-only stack (GPT, Llama) has only two.
60 second bullets to scan on the way to the call.
The three sublayers an encoder-decoder decoder block contains and their order
Where Q, K, and V come from in self-attention vs cross-attention
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.