The context vector for token i is the weighted sum of V rows, where the weights are row i of softmax(QK^T / sqrt(d_k)); it is the per-token attention output before W_O.
Imagine you ask a librarian for help with a question. The librarian glances at every book on the shelf and gives each one a probability of being relevant (these are the attention weights). Then she does not just hand you the most relevant book; she mixes a custom summary by taking a little bit from each book, weighted by relevance, and writes you one paragraph. That custom summary is the context vector: one tailor-made answer for your specific question, built from the whole shelf in proportion to relevance.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The 'context vector' is one of the most confused pieces of attention vocabulary. It sits at a specific point in the attention pipeline, between the softmax and the output projection, and confusing it with any of the neighboring quantities (Q, K, A, O) is one of the most common interview-grade mistakes.
This card walks the full attention pipeline in matrix form, labels every intermediate quantity, and pins down the context vector's exact location and dimensionality. It then connects this to two production-relevant viewpoints: how FlashAttention computes context vectors without materializing the attention weight matrix, and how the residual-stream view of mechanistic interpretability treats the context vector as the bridge between the QK circuit and the OV circuit.
By the end you should be able to point to any line of attention code and say exactly which intermediate quantity it produces.
The full attention pipeline in matrix form
Single-head scaled dot-product attention can be written as five sequential matrix operations.
The pipeline
Input: X (seq_len, d_model)
Step 1: Q = X W_Q (seq_len, d_k)
K = X W_K (seq_len, d_k)
V = X W_V (seq_len, d_v)
Step 2: S = Q K^T / sqrt(d_k) (seq_len, seq_len) raw scores
Step 3: A = softmax(S, axis=-1) (seq_len, seq_len) attention weights
Step 4: C = A . V (seq_len, d_v) CONTEXT VECTORS
Step 5: O = C W_O (seq_len, d_model) layer output
What each quantity means
X: the input embedding stream for this layer.Q, K, V: linear projections of X. Q is the query (what the token wants to know), K is the key (the addressable identity of each token), V is the value (the content each token carries).S: raw dot-product scores per (query, key) pair. Scaled by1/sqrt(d_k)to keep variance under control.A: post-softmax attention weights. Rowiis a probability distribution over key positions, telling you how much tokeniattends to each other token.C: the context vectors. Rowiis the weighted sum of V rows using rowiof A as weights. This is the actual output of the attention computation, before any further projection.O: the layer output after projecting C throughW_O. This is what gets added back to the residual stream.
Where the context vector lives
Step 4. Output of A . V. One per query position, dimensionality d_v, shape (seq_len, d_v). NOT the attention weights from step 3 (those are scalars per pair, not vectors). NOT the layer output from step 5 (that has been projected through W_O and lives in d_model space).
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Object | Shape | Role |
|---|---|---|
| Q (queries) | (seq_len, d_k) | Input to attention, asks the question per token |
| K (keys) | (seq_len, d_k) | Input to attention, addressable identities |
| V (values) | (seq_len, d_v) | Input to attention, content being retrieved |
| A (attention weights) | (seq_len, seq_len) | Mixing distribution: row i is a prob distribution over keys |
| C (context vectors) | (seq_len, d_v) | Output of attention proper: row i is the mixed retrieval for token i |
| O (layer output) | (seq_len, d_model) | Post-W_O projection, added back to the residual stream |
Real products, models, and research that use this idea.
- FlashAttention computes context vectors tile by tile without ever materializing the full attention weight matrix A, shipping in production stacks for GPT-5.5, Claude Opus 4.7, and Llama 4 Maverick.
- Cross-attention in T5 produces context vectors in the encoder's V-space, mixed by attention weights derived from decoder queries against encoder keys.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the context vector in V-space (dim d_v) and not in Q-space or K-space (dim d_k)?
The attention output is built by mixing rows of V with the attention-weight distribution. The mixing operation preserves the dimensionality of V, not of Q or K. In practice d_k = d_v in most transformers (both equal d_head = d_model / n_heads), so the distinction is invisible at the implementation level, but conceptually the context vector inherits its dimensionality from V because that is what is being averaged.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing the context vector with the attention weights themselves. The weights are the mixing recipe (probabilities); the context vector is the mixed result (a vector in V-space).
60 second bullets to scan on the way to the call.
Where the context vector sits in the attention pipeline: after softmax, before W_O
Why the context vector is in V-space (dim d_v), not in K-space or Q-space
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.