Explain in plain language what Q, K, and V each represent in attention. Use an analogy if it helps.
Explain what Q (Query), K (Key), and V (Value) each represent in the attention mechanism. Why does the model need all three projections rather than just one?
Q is what each token is looking for, K is the indexable description each token offers, V is the content it contributes when matched. Splitting K from V decouples findability from usefulness.
Imagine walking into a library. You hold a slip with your topic written on it, that's the Query. Each book on the shelf has a small catalog card pinned to its spine listing its subjects and a one line summary, those cards are the Keys. The thick book behind each card is the Value. You scan the cards (not the books) to decide what's relevant, then you actually read the matching books. The card and the book serve different jobs: the card is short and designed to be findable, the book is long and designed to inform. Attention works the same way: each token publishes a short Key so other tokens can find it, and keeps its rich Value to contribute once chosen.
Detailed answer & concept explanation~5 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Walk the library analogy, write the scaled dot product formula with Q/K/V identified, explain the K-vs-V findability/usefulness split, derive why d_q = d_k while d_v can differ, and connect to how GQA/MQA exploit the K/V pairing for serving efficiency.
| Role | Semantic job | Optimization objective | Lives in |
|---|---|---|---|
| Q (Query) | What the current token wants to know | Align with K of relevant tokens | d_k-dim query space |
| K (Key) | Indexable description each token offers | Be findable by relevant queries | d_k-dim key space (same as Q) |
| V (Value) | Payload contributed when matched | Carry useful downstream signal | d_v-dim value space (can differ) |
Real products, models, and research that use this idea.
- The original Vaswani 2017 transformer gives each head its own W_Q, W_K, W_V matrices and concatenates head outputs through W_O.
- Llama 4 Maverick and Gemma 4 use grouped-query attention: groups of heads share one K/V pair while every head keeps its own Q, exploiting that Q is the diverse questioner.
- Multi-query attention in Falcon and PaLM goes further with a single K/V shared across all heads, trading a small quality dip for large KV-cache savings.
- ALBERT experimented with cross-layer parameter sharing of Q/K/V projections but kept the three way split within each layer.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat goes wrong if you tie K and V into a single projection in practice?
QWhy do GQA and MQA share K/V across heads but never share Q?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Saying 'Q, K, V are just three views of the same input' is mechanically true but misses the point: the three semantic roles are why three separate learned projections exist in the first place.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.