Inside a T5 style encoder-decoder, which sub-layer reads the encoder outputs, and through which projection?
The decoder's cross-attention sub-layer reads the encoder outputs: Q is projected from the decoder hidden state; K and V are projected from the encoder's final-layer hidden states.
Picture a translator with a notebook open to a page of English. The translator's pen is moving across a new page in French. Every few seconds the translator's eyes flick to the English page and pull a phrase, then write the equivalent in French. The pen position and what they want to write are the query. The English notebook is what they read from. Cross-attention is exactly this: the decoder asks the question (Q from its own state), and the encoder's page is what it reads (K and V from the encoder). The two stacks meet at this one sub-layer.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Cross-attention is the sub-layer that distinguishes encoder-decoder transformers from every other transformer family. It is also the only place in the architecture where information flows between two distinct sub-networks. Understanding exactly how it works, which projection matrices it uses, where its Q, K, V come from, and why it is not causally masked, is the foundation for understanding why encoder-decoder is structurally different from decoder-only, why translation services serve N target languages from one source efficiently, and why hybrid architectures like Flamingo can fuse vision encoders into language models.
This deep dive walks the precise mechanics, contrasts cross-attention with the other two sub-layers in a T5-style decoder block, surveys the serving implications, and closes with the variants that exploit the asymmetric Q-from-decoder, K-and-V-from-encoder geometry.
The three sub-layers of a T5 decoder block
A T5-style decoder block has three sub-layers in a fixed order, each wrapped in pre-norm and residual connections.
Sub-layer 1: masked self-attention
The decoder attends to its own past tokens (positions less than or equal to the current position). Q, K, V are all projected from the decoder's incoming hidden state through self-attention W_Q, W_K, W_V. A causal mask enforces the 'past only' constraint.
Sub-layer 2: cross-attention
This is where the decoder reads from the encoder. Q is projected from the decoder hidden state (output of sub-layer 1 after residual + norm) through cross-attention W_Q. K and V are projected from the encoder's final-layer hidden states through dedicated cross-attention W_K and W_V matrices. No causal mask is applied, the encoderside is fully known.
Sub-layer 3: FFN
The standard per-token MLP, identical to what an encoder block uses. Two linear layers with an activation in between.
The block in pseudocode
def decoder_block(x, enc_out):
# 1. Masked self-attention
h = self_attn(norm(x), causal_mask=True)
x = x + h
# 2. Cross-attention (reads from encoder)
h = cross_attn(query=norm(x), key_value_source=enc_out)
x = x + h
# 3. FFN
h = ffn(norm(x))
x = x + h
return x
The cross-attention sub-layer is sandwiched between the decoder's self-attention and its FFN. That position matters: the query reflects what the decoder already knows about its own context, after the self-attention pass.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- T5 (Google, 2019): the canonical encoder-decoder; every decoder block has masked self-attention, cross-attention, FFN.
- BART (Facebook, 2019): same three sub-layer decoder block as T5; cross-attention reads the encoder's bidirectional output.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf every decoder block reads from the same encoder output, why does each block need its own W_K and W_V?
Different decoder layers need to read different aspects of the encoder representation. Bottom decoder layers might attend to surface tokens; deeper layers to semantic structure. Each layer's W_K and W_V learn the projection appropriate for that layer's reading needs. Sharing them would force every layer to read the same features.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Believing the encoder and decoder share QKV projection matrices. Cross-attention uses dedicated W_K and W_V matrices separate from the decoder's self-attention projections.
60 second bullets to scan on the way to the call.
Three sub-layers in a T5-style decoder block, in order
Cross-attention sub-layer position (between self-attention and FFN)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.