Explain the difference between self-attention and cross-attention. When does each appear in a real architecture?

Short answer·Medium·4.0 · 0·~3 min·Asked atAda Fireworks Ai Uber·Relevant atMicrosoft

Attempt it

Explain the difference between self-attention and cross-attention in terms of where Q, K, V come from. When does each appear in a real architecture? Give concrete examples (BERT, GPT, T5).

Free · 2 AI evals / day

The whole difference between self-attention and cross-attention is where Q, K, and V (query, key, value) come from. In self-attention, all three projections read from the same input x: `q = W_Q x`, `k = W_K x`, `v = W_V x`. Each token attends to other tokens in its own sequence. There are sub-flavors based on the mask, BERT-style bidirectional or GPT-style causal, but the routing is the same. In cross-attention, Q is projected from one source and K, V are projected from a different source. The classic pattern is encoder-decoder translation. The encoder reads the source sentence and produces a sequence of hidden states. The decoder generates the target sentence while cross-attending to those hidden states. So `q = W_Q x_dec`, `k = W_K x_enc`, `v = W_V x_enc`. Each decoder position can attend to all encoder positions. Real architectures map cleanly to families. BERT, RoBERTa, and DeBERTa are encoder-only and use bidirectional self-attention. GPT, Llama, and Mistral are decoder-only and use causal self-attention; they have no encoder, so no cross-attention. T5, BART, and Whisper are encoder-decoder, where each decoder block has three sublayers: causal self-attention, cross-attention to the encoder, then an MLP.

It depends on where Q, K, and V get their input. That one routing fact is the whole difference. In self-attention, all three come from the same sequence x. Q asks, K advertises, V delivers, and the conversation happens within one stream. BERT does this bidirectionally, GPT and Llama do it causally, and the encoder side of T5 does it bidirectionally too. In cross-attention, Q comes from one sequence and K/V come from a different one. The standard case is the T5 decoder. Each block does causal self-attention over the tokens generated so far, then cross-attention with Q from the decoder and K/V from the encoder output, then the MLP. That cross-attention is what lets the decoder ground its generation in the source. One practical consequence: the cross-attention K/V cache is computed once when the encoder runs and reused across every decode step. The self-attention cache grows by one row per generated token. So encoder-decoder inference for fixed input tasks is pretty efficient: the expensive cache is only the self-attention one. Decoder-only LLMs skip cross-attention because they prepend everything (instructions, context, query) into one sequence and let causal self-attention handle it all uniformly. That's the architectural simplification that made modern serving stacks work at scale.

TL;DR

Self-attention: Q, K, V from the same sequence (BERT, GPT, Llama). Cross-attention: Q from one sequence, K/V from another (T5, BART decoders). Decoder-only LLMs skip cross-attention because they have no separate encoder.

Memory aid

Easy to grasp

Imagine taking notes during a lecture. Self-attention is like glancing back at your own notes as you write the next line: one notebook, one stream. Cross-attention is like a translator at a desk: they look at a textbook open on their left while writing on a fresh page on their right. The pen on the right page is the Query; the textbook on the left is what they're keying and valuing against. Decoder-only models like GPT and Llama are the note-taker with one notebook. T5 and BART are the translator with two documents in play.

Key concepts

Self-attention Cross-attention Encoder-decoder architectures Q/K/V routing

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Self-attention and cross-attention differ entirely in where Q, K, and V come from. That one routing fact determines which architectural family a model belongs to. Get the routing right and BERT, GPT, T5, and Whisper sort themselves out.

Mental model: count the towers. Same tower architectures use only self-attention. Two tower architectures use both self-attention within each tower and cross-attention between them.

The Q/K/V routing, explicitly

Self-attention

One input tensor x, three learned projections:

q = W_Q x
k = W_K x
v = W_V x

The attention pattern is T × T. Each token attends to other tokens in the same sequence, subject to a mask (bidirectional in BERT, causal in GPT/Llama).

Cross-attention

Two input tensors, asymmetric routing:

q = W_Q x_dec from the decoder side
k = W_K x_enc and v = W_V x_enc from the encoder side

The attention pattern is T_dec × T_enc, non-square when the sequence lengths differ (which they almost always do).

Why the direction matters

The convention is fixed: Q is from the decoder, K and V are from the encoder. Reversing this (encoder-Q, decoder-K/V) would mean the encoder attends to the decoder. That is the wrong direction for grounding generation in the source.

The asymmetric routing is what turns 'attention' into 'cross-attention'. Same routing under both Q and K/V is plain self-attention with two streams concatenated.

Where each appears in real architectures

The production deployment lens

Where two tower cross-attention still wins

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Aspect	Self-attention	Cross-attention
Q source	Same sequence x	Decoder sequence x_dec
K, V source	Same sequence x	Encoder sequence x_enc
Attention matrix shape	T × T (square)	T_dec × T_enc (non-square)
KV cache during decoding	Grows one row per token	Fixed once encoder runs
Production deployment	Same tower, one cache layout	Two tower, dual cache regimes
Used in	BERT, GPT, Llama, all encoders	T5/BART decoder, Whisper, Flamingo

Real products, models, and research that use this idea.

BERT, RoBERTa, DeBERTa: encoder-only with bidirectional self-attention. No cross-attention anywhere.
GPT-3, Llama 4 Maverick, Mistral Large 3, DeepSeek-V3: decoder-only with causal self-attention only.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy doesn't the cross-attention KV cache grow across decoding steps the way self-attention's does?

K and V in cross-attention are projections of the encoder output, which is computed once and then held fixed for the entire generation. Each decode step uses a new Q but the same cached K and V. Self-attention's cache must grow because each newly generated token contributes a new K and V row.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Reversing the direction of cross-attention. In encoder-decoder cross-attention, Q is from the decoder and K/V are from the encoder, not the other way around.

60 second bullets to scan on the way to the call.

Q/K/V routing for self vs cross-attention
Direction of Q vs K/V flow in cross-attention

Primary sources. Browse if you want the original framing.