The residual stream is the d_model-wide per-token vector that flows untouched through the network; every sub-layer reads it via projections and writes a delta back via a skip add.
Picture a long conveyor belt running through a factory. At every station, a worker reaches over, looks at what is on the belt, builds a small modification, and drops the modification back onto the belt without removing the original. By the time the conveyor reaches the end, the product carries the original input plus every worker's contribution layered on top. That conveyor belt is the residual stream. Attention is one type of worker; the feed-forward layer is another. They all read the belt and add to it, never replacing it.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The residual stream is one of the most important concepts in modern transformer interpretability and a great window into how transformer architecture actually works. It is not a fancy concept; it is just the unmodified per-token vector that flows through every block of the network. But framing it as a persistent signal bus (rather than as 'whatever variable x happens to be at this layer') unlocks a much cleaner mental model of how attention and FFN sub-layers interact across depth.
This deep dive starts with the concrete definition and shape of the stream, walks through how an attention sub-layer reads from and writes to it, contrasts pre-norm and post-norm in terms of stream preservation, explains the mechanistic-interpretability framing that treats sub-layers as agents communicating via the stream, and closes with the bandwidth-scaling consequences for very deep networks.
By the end you should be able to draw a transformer block as a conveyor belt with two stations and explain why every sub-layer in the network can communicate with every other one through that belt.
What the residual stream physically is
For a transformer with T input tokens, d_model hidden dimension, and L layers, the residual stream is a (T, d_model) tensor that exists at every layer boundary. The same (T, d_model) shape persists from after the input embedding all the way to before the final output projection.
How it starts
At input: x = embed(tokens) + PE(positions) for absolute-PE models, or just x = embed(tokens) for RoPE-style models (where positional information is injected inside attention rather than into the stream).
How it evolves
After every transformer block, the stream is updated by ADDING the block's contributions:
# Pre-norm block
x = x + attention(rmsnorm(x))
x = x + ffn(rmsnorm(x))
The stream now carries x_initial + attention_contribution + ffn_contribution after one block. After L blocks, it carries the initial embedding plus 2L sub-layer contributions (one from each attention sub-layer, one from each FFN sub-layer).
How it ends
At the output, the stream goes through a final RMSNorm and is projected to logits over the vocabulary:
logits = output_proj(rmsnorm(x)) # shape (T, vocab_size)
For each token, the prediction is a function of the full accumulated contributions to that token's residual stream vector.
Per-token independence
The residual streams for different tokens are independent in the sense that attention is the ONLY operation that mixes information across tokens. The FFN operates per-token, and the normalization is per-token. So you can think of the residual stream as one independent column per token, with attention being the cross-token gather operation.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic's transformer-circuits work models induction heads as residual-stream circuits where a head in one layer writes to subspaces a head in a later layer reads.
- Llama 4 Maverick maintains an 8192-wide residual stream across 80+ blocks; sub-layer contributions accumulate before the final projection.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you read out the contribution of a specific attention head to the residual stream at a given layer?
Decompose the layer's attention into per-head terms W_{O,h} @ attention_h(LN(x)) @ W_{V,h} and look at the specific subspace W_{O,h} writes to. Linear probing or activation patching can isolate the contribution of head h to the stream at that layer.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking each layer REPLACES the activations rather than ADDS to them. The whole architectural point is that information persists down the stack via the unmodified skip path.
60 second bullets to scan on the way to the call.
Define the residual stream as the d_model-wide per-token vector flowing through the network
Describe how Q, K, V are projections of the (normalized) stream
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.