The residual stream is the d_model-wide per-token vector that flows untouched through the network. Every sub-layer reads it via projections and writes a delta back via a skip add.
Picture a long conveyor belt running through a factory. At every station, a worker reaches over and looks at what is on the belt. The worker builds a small modification and drops it back onto the belt without removing the original. By the time the conveyor reaches the end, the product carries the original input plus every worker's contribution layered on top. That conveyor belt is the residual stream. Attention is one type of worker; the feed-forward layer is another. They all read the belt and add to it, never replacing it.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The residual stream is one of the most important concepts in modern transformer interpretability and a great window into how transformer architecture works. It is not a fancy concept. It is just the unmodified per token vector that flows through every block of the network. But you can frame it as a persistent signal bus (rather than as 'whatever variable x happens to be at this layer'). This framing unlocks a much cleaner mental model of how attention and FFN sub-layers interact across depth.
You can draw a transformer block as a conveyor belt with two stations. Every sub-layer in the network can communicate with every other one through that belt.
What the residual stream physically is
For a transformer with T input tokens, d_model hidden dimension, and L layers, the residual stream is a (T, d_model) tensor that exists at every layer boundary. The same (T, d_model) shape persists from after the input embedding all the way to before the final output projection.
How it starts
At input: x = embed(tokens) + PE(positions) for absolute-PE models, or just x = embed(tokens) for RoPE-style models (where positional information is injected inside attention rather than into the stream).
How it evolves
After every transformer block, the stream is updated by ADDING the block's contributions:
# Pre-norm block
x = x + attention(rmsnorm(x))
x = x + ffn(rmsnorm(x))
The stream now carries x_initial + attention_contribution + ffn_contribution after one block. After L blocks, it carries the initial embedding plus 2L sub-layer contributions (one from each attention sub-layer, one from each FFN sub-layer).
How it ends
At the output, the stream goes through a final RMSNorm and is projected to logits over the vocabulary:
logits = output_proj(rmsnorm(x)) # shape (T, vocab_size)
For each token, the prediction is a function of the full accumulated contributions to that token's residual stream vector.
Independence across tokens
The residual streams for different tokens are independent in the sense that attention is the ONLY operation that mixes information across tokens. The FFN operates per token, and the normalization is per-token. So you can think of the residual stream as one independent column per token, with attention being the cross-token gather operation.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's transformer-circuits work models induction heads as residual-stream circuits where a head in one layer writes to subspaces a head in a later layer reads.
- Llama 4 Maverick maintains a 5120-wide residual stream across 48 blocks; sub-layer contributions accumulate before the final projection.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you read out the contribution of a specific attention head to the residual stream at a given layer?
Decompose the layer's attention into per-head terms W_{O,h} @ attention_h(LN(x)) @ W_{V,h} and look at the specific subspace W_{O,h} writes to. Linear probing or activation patching can isolate the contribution of head h to the stream at that layer.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking each layer REPLACES the activations rather than ADDS to them. The whole architectural point is that information persists down the stack via the unmodified skip path.
60 second bullets to scan on the way to the call.
Define the residual stream as the d_model-wide per-token vector flowing through the network
Describe how Q, K, V are projections of the (normalized) stream
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.