Describe the residual stream as a read/write surface that sublayers act on
The residual stream is a shared activation bus, shape (batch, seq_len, d_model), that flows through every block. Each sublayer reads a normalized snapshot of it and adds a delta back.
Picture a long roll of carbon paper that gets passed down an assembly line. Every station gets a copy of what is currently on the paper (the 'read'), works on its copy, and then stamps a small correction back onto the original paper (the 'write'). The paper itself is never erased; it just accumulates more and more refinements as it moves through the line. In a transformer, that carbon paper is the residual stream. The stations are the attention and feed-forward sublayers, and 'read a copy' is what the LayerNorm does at the start of each sublayer. Eighty stations later, the paper carries every refinement the model wanted to make.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The residual stream is the single most useful mental model for what a transformer is doing internally. Once you have it, the block structure stops feeling like a chain of transformations and starts feeling like a chorus of small contributions onto a shared running representation. This question is asking whether you have the right mental model, not whether you can recite the block recipe.
What the residual stream actually is
The residual stream is a tensor of shape (batch, seq_len, d_model) that starts as the input embedding (token embedding plus positional information) and flows through every block of the stack. Its shape does not change between blocks. The same d_model-dimensional vector at position i that exists after block 1 still exists, modified additively, after block 80.
This is unusual among neural network architectures. Convolutional nets reshape their activations between layers (different channel counts, different spatial resolutions). RNNs carry a hidden state but it is also typically transformed by each step. Transformers deliberately keep the activation shape fixed so that every block can read and write to a common surface.
Where it comes from. Input embedding lookup produces a tensor of shape (batch, seq_len, d_model). If positional information is added (sinusoidal, learned, or written by a position-aware first sublayer), it lands on this same tensor. From that moment until the final unembedding head, this tensor IS the residual stream.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Convolutional net | Transformer (residual stream) |
|---|---|---|
| What flows between layers | A transformed activation (replaced each layer) | A running residual tensor (added to each block) |
| Shape between layers | Can change (channels, spatial dims) | Constant at `(batch, seq_len, d_model)` |
| What 'depth' means | Sequential transformation | Additive refinement on a shared bus |
| Information from layer 0 | Often lost or compressed | Persists structurally to the top |
Real products, models, and research that use this idea.
- Anthropic's 'A Mathematical Framework for Transformer Circuits' (2021) introduced the residual-stream-as-bus framing that became standard in mech-interp.
- Llama 3.1 8B carries a residual stream of shape `(batch, seq_len, 4096)` through 32 blocks; the same tensor is the input to the final RMSNorm and unembedding head.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf the residual stream is just `x + delta_1 + delta_2 + ...`, why do interpretability researchers care about which sublayer wrote which delta?
Because each delta carries the signature of its source sublayer's weights and inputs; tracing which sublayer wrote a feature is what lets you build circuits across blocks.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Saying the sublayers 'replace' the activation. They never do. Every sublayer adds a delta onto the residual stream; the original information persists by construction.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.