Identify the context vector inside scaled dot-product attention and what produces it.
The context vector for token i is the weighted sum of V rows, where the weights are row i of softmax(QK^T / sqrt(d_k)); it is the per-token attention output before W_O.
Imagine you ask a librarian for help with a question. The librarian glances at every book on the shelf and gives each one a probability of being relevant (these are the attention weights). Then she does not just hand you the most relevant book; she mixes a custom summary by taking a little bit from each book, weighted by relevance, and writes you one paragraph. That custom summary is the context vector: one tailor-made answer for your specific question, built from the whole shelf in proportion to relevance.
Detailed answer & concept explanation~6 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: vocabulary walk through Q, K, V, A, C, O; where the context vector sits in the pipeline; per-head vs per-layer ambiguity; how FlashAttention computes context vectors without materializing A; role in the residual stream view.
| Object | Shape | Role |
|---|---|---|
| Q (queries) | (seq_len, d_k) | Input to attention, asks the question per token |
| K (keys) | (seq_len, d_k) | Input to attention, addressable identities |
| V (values) | (seq_len, d_v) | Input to attention, content being retrieved |
| A (attention weights) | (seq_len, seq_len) | Mixing distribution: row i is a prob distribution over keys |
| C (context vectors) | (seq_len, d_v) | Output of attention proper: row i is the mixed retrieval for token i |
| O (layer output) | (seq_len, d_model) | Post-W_O projection, added back to the residual stream |
Real products, models, and research that use this idea.
- FlashAttention computes context vectors tile by tile without ever materializing the full attention weight matrix A, shipping in production stacks for GPT-5.5, Claude Opus 4.7, and Llama 4 Maverick.
- Cross-attention in T5 produces context vectors in the encoder's V-space, mixed by attention weights derived from decoder queries against encoder keys.
- Mechanistic interpretability work on Claude and Gemini decomposes the OV circuit (V W_O) separately from the QK circuit, treating the context vector as the bridge between attention pattern and residual stream contribution.
- Mixture of experts transformers like DeepSeek V4 retain the standard attention block structure, with context vectors flowing into expert routing rather than to a dense FFN.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the context vector in V-space (dim d_v) and not in Q-space or K-space (dim d_k)?
QHow does FlashAttention compute context vectors without materializing the full attention weight matrix?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Confusing the context vector with the attention weights themselves. The weights are the mixing recipe (probabilities); the context vector is the mixed result (a vector in V-space).
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.