In self-attention, where do Q, K, and V come from? In cross-attention?

Attempt it

Quick reference

TL;DR

Self-attention: Q, K, V all from the same input via separate linear projections. Cross-attention: Q from one stream (decoder), K and V both from the other (encoder).

Memory aid

Easy to grasp

Imagine looking something up in a dictionary. You arrive with a question: that's the query. The dictionary has page tabs you flip through to find a match, those are the keys. When you find the right tab, the actual definition you read is the value. The tab and the definition always belong to the same dictionary, but the question you arrived with can come from anywhere: your own head, or a friend handing you a note.

Key concepts

Query/Key/Value projections Self-attention Cross-attention

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define the three projections, motivate the dictionary-lookup analogy, explain the K/V pairing rationale, walk the gradient-flow story for why they aren't tied, and distinguish self versus cross from masking with T5/BART/Whisper for cross and Llama/GPT for self-only.

Code example

python

import torch.nn as nn

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)
        self.n_heads = n_heads

    def forward(self, x_q, x_kv):
        # Self-attention: caller passes x_q = x_kv = x
        # Cross-attention: caller passes x_q = decoder_h, x_kv = encoder_out
        Q = self.W_q(x_q)
        K = self.W_k(x_kv)
        V = self.W_v(x_kv)
        # ... split heads, scaled dot product, recombine ...
        return out

Side-by-side

Block type	Source of Q	Source of K	Source of V
Encoder self-attention	Encoder input X	Encoder input X	Encoder input X
Decoder self-attention (causal)	Decoder hidden D	Decoder hidden D	Decoder hidden D
Cross-attention	Decoder hidden D	Encoder output E	Encoder output E

Real-world examples

Real products, models, and research that use this idea.

T5 is the canonical encoder decoder model with cross-attention in every decoder layer.
Whisper uses cross-attention in the text decoder to attend over encoded audio features.
Flamingo inserts cross-attention layers into a frozen language model to attend over visual features.
Claude Opus 4.7, GPT-5.5, and Llama 4 Maverick are decoder-only and have NO cross-attention: only causal self-attention.
BERT is encoder-only with only bidirectional self-attention; modern encoder revivals (RetroMAE, BGE-M3) follow the same pattern.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy don't decoder-only models like Llama 4 need cross-attention even though they handle conditioning (RAG context, system prompts)?

QIn multi-modal models like Flamingo, why insert cross-attention rather than retraining a unified self-attention stream?

QWhat goes wrong if you tie W_Q, W_K, and W_V to a single shared matrix?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing self-attention masking (causal vs bidirectional) with the self vs cross distinction. Masking and Q/K/V sourcing are orthogonal axes.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.