Explain the difference between self-attention and cross-attention in terms of where Q, K, V come from. When does each appear in a real architecture? Give concrete examples (BERT, GPT, T5).
Free · 2 AI evals / day
The whole difference between self-attention and cross-attention is where Q, K, and V come from. Get that clear and the architectures fall out automatically.
In self-attention, all three projections read from the same input x: `q = W_Q x`, `k = W_K x`, `v = W_V x`. Each token attends to other tokens in its own sequence. There are sub-flavors based on the mask, BERT-style bidirectional or GPT-style causal, but the routing is the same.
In cross-attention, Q is projected from one source and K, V are projected from a different source. The classic pattern is encoder decoder translation: the encoder reads the source sentence and produces a sequence of hidden states, and the decoder generates the target sentence while cross-attending to those encoder hidden states. So `q = W_Q x_dec`, `k = W_K x_enc`, `v = W_V x_enc`. Each decoder position can attend to all encoder positions.
Real architectures map cleanly to families. BERT, RoBERTa, and DeBERTa are encoder-only and use bidirectional self-attention. GPT, Llama, Mistral, and Claude are decoder-only and use causal self-attention; they have no encoder, so no cross-attention. T5, BART, and Whisper are encoder decoder, where each decoder block has three sublayers: causal self-attention, cross-attention to the encoder, then an MLP.
Depends on where Q, K, and V get their input, that one routing fact is the whole difference.
In self-attention, all three come from the same sequence x. Q asks, K advertises, V delivers, and the conversation happens within one stream. BERT does this bidirectionally, GPT and Llama do it causally, and the encoder side of T5 does it bidirectionally too.
In cross-attention, Q comes from one sequence and K/V come from a different one. The canonical case is the T5 decoder: each block does causal self-attention over the tokens generated so far, then cross-attention with Q from the decoder and K/V from the encoder output, then the MLP. That cross-attention is what lets the decoder ground its generation in the source.
One practical consequence worth naming: the cross-attention K/V cache is computed once when the encoder runs and reused across every decode step. The self-attention cache grows by one row per generated token. So encoder decoder inference for fixed input tasks is actually pretty efficient: the expensive cache is only the self-attention one. Decoder-only LLMs skip cross-attention because they prepend everything (instructions, context, query) into one sequence and let causal self-attention handle it all uniformly. That's the architectural simplification that made modern serving stacks work at scale.
Quick reference
TL;DR
Self-attention: Q, K, V from the same sequence (BERT, GPT, Llama). Cross-attention: Q from one sequence, K/V from another (T5, BART decoders). Decoder-only LLMs skip cross-attention because they have no separate encoder.
Imagine taking notes during a lecture. Self-attention is like glancing back at your own notes as you write the next line: one notebook, one stream. Cross-attention is like a translator at a desk: they look at a textbook open on their left while writing on a fresh page on their right. The pen on the right page is the Query; the textbook on the left is what they're keying and valuing against. Decoder-only models like GPT and Llama are the note-taker with one notebook. T5 and BART are the translator with two documents in play.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Building a general-purpose 2026 chat LLM stack Decoder-only is the production-default for serving simplicity; adding cross-attention forces you to maintain two attention regimes and dual KV cache layouts for no quality win.
Latency-sensitive streaming where input grows alongside output Cross-attention assumes the encoder output is fixed before decoding starts; growing-input workloads have no stable encoder representation to attend to.
Encoder-only embedding workloads (BERT-style retrieval, classification) These need bidirectional self-attention to produce a single vector per input; cross-attention requires a second sequence that does not exist in this setup.
Single-stream autoregressive generation with no separate input modality If everything you want the model to consider is already in one sequence, prepending it under causal self-attention is strictly simpler than introducing a separate encoder.
Quick scan · 2–4 min
Everything important, quickly.
Walk the Q/K/V routing distinction with explicit equations, map BERT/GPT/T5/Whisper to their families, describe the T5 decoder block's three sublayers, explain the same tower vs two tower production analogy, and cover Flamingo-style cross-modal injection as the modern use case.
Side-by-side
Aspect
Self-attention
Cross-attention
Q source
Same sequence x
Decoder sequence x_dec
K, V source
Same sequence x
Encoder sequence x_enc
Attention matrix shape
T × T (square)
T_dec × T_enc (non-square)
KV cache during decoding
Grows one row per token
Fixed once encoder runs
Production deployment
One-tower, one cache layout
Two tower, dual cache regimes
Used in
BERT, GPT, Llama, all encoders
T5/BART decoder, Whisper, Flamingo
Real-world examples
Real products, models, and research that use this idea.
1BERT, RoBERTa, DeBERTa: encoder-only with bidirectional self-attention. No cross-attention anywhere.
2GPT-5.5, Claude Opus 4.7, Llama 4 Maverick, Gemini 3.1 Pro, DeepSeek V4: decoder-only with causal self-attention only.
3T5 and BART: encoder decoder where the encoder uses bidirectional self-attention and each decoder block has causal self-attention, cross-attention to the encoder, and an MLP.
4Whisper: encoder decoder for speech to text where the text decoder cross-attends to the audio encoder output.
5Flamingo and modern VLMs: gated cross-attention layers inserted into a frozen LM to inject vision features without retraining the language backbone.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy doesn't the cross-attention KV cache grow across decoding steps the way self-attention's does?
Sign in or go Pro to see the approach for this follow-up.
QWhy is the cross-attention matrix non-square as T_dec × T_enc?
Sign in or go Pro to see the approach for this follow-up.
QIn Flamingo, what does 'gated' cross-attention mean and why is the gate needed?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reversing the direction of cross-attention. In encoder decoder cross-attention, Q is from the decoder and K/V are from the encoder, not the other way around.