Inside a T5-style encoder-decoder, which sub-layer reads the encoder outputs, and through which projection?

Attempt it

Quick reference

TL;DR

The decoder's cross-attention sub-layer reads the encoder outputs: Q is projected from the decoder hidden state; K and V are projected from the encoder's final-layer hidden states.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a translator with a notebook open to a page of English. The translator's pen is moving across a new page in French. Every few seconds the translator's eyes flick to the English page and pull a phrase, then write the equivalent in French. The pen position and what they want to write are the query. The English notebook is what they read from. Cross-attention is exactly this: the decoder asks the question (Q from its own state), and the encoder's page is what it reads (K and V from the encoder). The two stacks meet at this one sub-layer.

Key concepts

Cross-attention Encoder-decoder architecture Self-attention Q, K, V projections

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Name the three sub-layers in the decoder block, place cross-attention in the middle, identify Q from decoder and K, V from encoder, note the dedicated cross-attention W_K and W_V per layer, and close with cross-attention K, V caching as the serving win.

Real-world examples

Real products, models, and research that use this idea.

T5 (Google, 2019): the canonical encoder-decoder; every decoder block has masked self-attention, cross-attention, FFN.
BART (Facebook, 2019): same three sub-layer decoder block as T5; cross-attention reads the encoder's bidirectional output.
Whisper (OpenAI ASR, 2022): audio encoder → text decoder; cross-attention is the bridge, and its K, V are cached across all beam-search hypotheses.
Flamingo (DeepMind, 2022): decoder-only LLM augmented with cross-attention layers that read a frozen vision encoder's output.
Hugging Face's `T5ForConditionalGeneration`: implements all three decoder sub-layers explicitly; you can inspect the cross-attention weights via `output_attentions=True`.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf every decoder block reads from the same encoder output, why does each block need its own W_K and W_V?

Sign in or go Pro to see the approach for this follow-up.

QWhy is cross-attention not causally masked?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing the encoder and decoder share QKV projection matrices. Cross-attention uses dedicated W_K and W_V matrices separate from the decoder's self-attention projections.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.