Walk through a T5 decoder block in order, list the attention sub-layers and what each one's Q, K, V read from.

Order steps·Medium·4.0 · 0·~1 min·Asked atCursor Goldman Sachs Hugging Face·Relevant atAi4bharat Cerebras Deepseek Microsoft

Attempt it

1Decoder self-attention: Q, K, V all projected from the decoder's own current hidden state, with a causal lower triangular mask so position t only sees positions <= t.
2Feed-forward sub-layer (FFN): operates per token, no cross-token interaction. Runs on the residual stream and pushes its output back into the residual.
3Cross-attention: Q projected from the decoder hidden state, while K and V are projected from the final layer encoder output. No causal mask on the encoder axis, the encoder is fully observed.

**The correct order is: causal self-attention, then cross-attention, then position-wise FFN.** A T5 (and original Transformer 2017) decoder block has three sub-layers, each with its own residual connection and layer norm. ### Sub-layer 1: causal self-attention The decoder looks at its own previously-generated tokens. Q, K, and V are all projected from the decoder's current hidden state. A causal lower-triangular mask ensures position `t` can only attend to positions `0, 1, ..., t`, not to future positions. This is what makes generation autoregressive. ### Sub-layer 2: cross-attention (the encoder-decoder bridge) The decoder consults the encoder's output. Q is projected from the decoder's hidden state (the updated state after self-attention). K and V are projected from the encoder's final-layer output, which is fixed across decoder steps. **No causal mask on the encoder axis** because the encoder is fully observed, the decoder can attend to any encoder position. This is where the encoder and decoder communicate. Cross-attention is what makes encoder-decoder models useful for translation, summarization, and any task with a source-target structure. ### Sub-layer 3: position-wise FFN A two-layer MLP applied independently to each position. No cross-token interaction. Refines each token's representation based on the combined self-attention and cross-attention output. ### Why this order The decoder must process its own context first (what has been generated so far) before forming a useful query against the encoder output. Reversing the order, doing cross-attention before self-attention, would force the cross-attention query to be based on the raw input embedding without any awareness of the decoder's own state. The order causal -> cross -> FFN is what enables coherent generation.

**The correct order is: causal self-attention, then cross-attention, then position-wise FFN.** A T5 decoder block has three sub-layers in this fixed sequence, each wrapped with residual + layer norm. ### Sub-layer 1: decoder self-attention - Q, K, V all projected from the decoder's current hidden state. - Causal lower-triangular mask: position `t` attends only to positions `0, 1, ..., t`. - Updates the decoder's representation based on its own previously-generated tokens. ### Sub-layer 2: cross-attention - Q projected from the decoder's hidden state (post self attention). - K and V projected from the **final-layer encoder output**, which is fixed across all decoder steps. - No causal mask on the encoder axis, the decoder can attend to any encoder position freely. - This is the encoder-decoder bridge. Without it, the decoder would have no access to the source sequence at all. ### Sub-layer 3: position-wise FFN - Two-layer MLP applied independently to each position. - No cross-token interaction. - Refines each token's representation. ### Why this order matters The decoder must process its own context (via causal self-attention) before issuing a query against the encoder output (via cross-attention). The cross-attention Q is built from the decoder's UPDATED hidden state, not its raw input. This lets the query be "what do I need from the source given what I have generated so far?" rather than "what does this raw position need from the source?" The FFN comes last because it has no information to contribute until both attention sub-layers have done their job of moving information between positions and across the encoder-decoder boundary. **Mnemonic**: review your work (self-attn) -> consult the source (cross-attn) -> polish each word (FFN).

TL;DR

T5 decoder block order: causal self-attention first (Q, K, V from decoder), then cross-attention (Q from decoder, K, V from encoder), then position-wise FFN, each with residual and norm.

Memory aid

Easy to grasp

Think of a translator writing the next word of a translation. First the translator reviews what they have already written so far (causal self-attention: looking at their own previous work). Then the translator looks at the original sentence to see what still needs translating (cross-attention: consulting the encoder). Finally the translator polishes the wording for the next word individually (FFN: per-token refinement). The order matters because reviewing your own progress before consulting the source means you ask the source the right question.

Key concepts

Encoder-decoder transformer Cross-attention Self-attention T5

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The T5 decoder block is the canonical encoder-decoder transformer building block, descended directly from the 2017 Transformer paper and used in T5, BART, Marian, Whisper, Voicebox, and most encoder-decoder models in the 2026 production stack. Its three sub layer structure (causal self-attention, cross-attention, FFN) and the fixed ordering of those sub-layers encode a deep insight about how a decoder should integrate information.

This card walks the three sub-layers in order, explains what each one's Q, K, V read from and why, justifies the ordering choice with a concrete linguistic argument, covers the caching efficiency that makes cross-attention practical, and surveys where encoder-decoder architectures still dominate in 2026 despite the rise of decoder-only LLMs.

Sub-layer 1: causal self-attention

The first sub-layer in every T5 decoder block is decoder self-attention with a causal mask. This is what gives the decoder its autoregressive property and lets it integrate information about its own previously-generated tokens.

Projections

Q_dec = H_dec . W_Q
K_dec = H_dec . W_K
V_dec = H_dec . W_V

All three are linear projections of the decoder's current hidden state stack H_dec. Different weight matrices, but all reading from the same source.

The causal mask

A lower-triangular mask ensures position t attends only to positions 0, 1, ..., t. The upper triangle holds -inf (or the framework's negative-infinity sentinel) and is added to the raw scores before softmax. This prevents position t from peeking at future tokens during training; at inference time, future tokens do not exist yet, so the mask is implicit.

What this sub-layer accomplishes

At the moment the decoder is generating token t, this sub-layer lets that position pull information from every previously-generated token (and its own position via the diagonal). The decoder's hidden state at position t after this sub-layer encodes "what have I generated so far, and how does the current position fit into that?"

KV cache during inference

During autoregressive decoding, the decoder self-attention's K and V grow by one row per decode step. Production stacks cache these incremental K and V additions in a per-request KV cache to avoid recomputing them from scratch every step. This is the standard KV cache that PagedAttention and continuous batching manage.

Sub-layer 2: cross-attention (the encoder-decoder bridge)

Sub-layer 3: position-wise feed-forward network

Why this ordering, and why encoder-decoder survives in 2026

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Sub-layer	Q from	K, V from	Mask
Decoder self-attention	Decoder hidden state	Decoder hidden state	Causal lower-triangular
Cross-attention	Decoder hidden state (post self attn)	Encoder final-layer output	None on encoder axis; optional padding mask
Position-wise FFN	N/A (no attention)	N/A (per-token MLP)	N/A

Real products, models, and research that use this idea.

T5 (Raffel et al. 2020) is the canonical text to text encoder-decoder model used as the architectural reference for this layout.
T5-v1.1 and T5-1.1-XXL retain this exact block structure with GeGLU activations and relative position bias.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does cross-attention K, V get cached across all decoder steps?

The encoder runs once over the source input and produces a fixed final-layer hidden state. K and V for cross-attention are linear projections of that fixed output, so they are also fixed across the entire decoding process. Computing them once and reusing them across hundreds of decoder steps is one of the major efficiency wins of encoder-decoder architectures. The only K, V that change per decoder step are those from the decoder's own self-attention, which still need a per-step KV cache.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Putting cross-attention before self-attention. The decoder must process its own context first (causal self-attention) before forming a query against the encoder output (cross-attention).

60 second bullets to scan on the way to the call.

Three sub-layers per T5 decoder block: causal self-attn, cross-attn, FFN
Decoder self-attention with causal lower-triangular mask

Primary sources. Browse if you want the original framing.