Self-attention: Q, K, V all from the same input via separate linear projections. Cross-attention: Q from one stream (decoder), K and V both from the other (encoder).
Imagine looking something up in a dictionary. You arrive with a question: that's the query. The dictionary has page tabs you flip through to find a match, those are the keys. When you find the right tab, the actual definition you read is the value. The tab and the definition always belong to the same dictionary, but the question you arrived with can come from anywhere: your own head, or a friend handing you a note.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The Q/K/V sourcing distinction is the architectural definition of self- versus cross-attention. Get this wrong and you'll confuse causal masking with cross-attention, or assume GPT has a cross-attention block: both frequent interview anti-patterns.
This deep dive takes an architectural lens. The Q, K, V projections are best understood as a learned soft addressable database lookup, with W_Q, W_K, W_V as separately learned routing functions. That framing makes the K/V pairing obvious, motivates why the three projections aren't tied, and explains why self versus cross is a sourcing decision rather than a mechanism decision.
The payoff is a clean mental model that survives contact with decoder-only LLMs, encoder decoder revivals, and multi-modal architectures.
Self-attention: the sequence queries itself
All three projections start from the same input X. Three independent learned weight matrices produce Q = X · W_Q, K = X · W_K, V = X · W_V. The sequence is querying itself.
Where this lives in practice
- Encoder self-attention in BERT-style models: bidirectional, no causal mask.
- Decoder self-attention in GPT, Llama, Claude, Gemini: causal mask applied.
- Both encoder and decoder self-attention in T5 and BART.
Every layer of every modern transformer has at least one self-attention block. The architectural variant (encoder versus decoder, bidirectional versus causal) is a mask choice, not a sourcing choice.
Self-attention is the workhorse primitive. If you understand it, the rest of the attention family is just rewiring inputs.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import torch.nn as nn
class MultiHeadAttention(nn.Module):
def __init__(self, d_model, n_heads):
super().__init__()
self.W_q = nn.Linear(d_model, d_model)
self.W_k = nn.Linear(d_model, d_model)
self.W_v = nn.Linear(d_model, d_model)
self.W_o = nn.Linear(d_model, d_model)
self.n_heads = n_heads
def forward(self, x_q, x_kv):
# Self-attention: caller passes x_q = x_kv = x
# Cross-attention: caller passes x_q = decoder_h, x_kv = encoder_out
Q = self.W_q(x_q)
K = self.W_k(x_kv)
V = self.W_v(x_kv)
# ... split heads, scaled dot product, recombine ...
return out| Block type | Source of Q | Source of K | Source of V |
|---|---|---|---|
| Encoder self-attention | Encoder input X | Encoder input X | Encoder input X |
| Decoder self-attention (causal) | Decoder hidden D | Decoder hidden D | Decoder hidden D |
| Cross-attention | Decoder hidden D | Encoder output E | Encoder output E |
Real products, models, and research that use this idea.
- T5 is the canonical encoder decoder model with cross-attention in every decoder layer.
- Whisper uses cross-attention in the text decoder to attend over encoded audio features.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy don't decoder-only models like Llama 4 need cross-attention even though they handle conditioning (RAG context, system prompts)?
Decoder-only models prepend the conditioning into the same input stream and rely on causal self-attention to do the equivalent work. Simpler architecturally, but you lose the ability to cache the encoder representation independently of the decoder, every decoder layer pays for re-attending to the conditioning at every step.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing self-attention masking (causal vs bidirectional) with the self vs cross distinction. Masking and Q/K/V sourcing are orthogonal axes.
60 second bullets to scan on the way to the call.
Q, K, V sourcing in self-attention
Q, K, V sourcing in cross-attention
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.