Describe the residual stream as a read/write surface that sublayers act on

Attempt it

The residual stream is the running activation tensor of shape (batch, seq_len, d_model) that flows from block to block. Every sublayer READS a normalized copy of it (via LayerNorm / RMSNorm) and WRITES an additive update back onto it (via the residual add). The stream itself is never overwritten. Depth becomes iterative refinement, with attention and FFN contributing deltas at each block.

Your answer

Quick reference

TL;DR

The residual stream is a shared activation bus, shape (batch, seq_len, d_model), that flows through every block. Each sublayer reads a normalized snapshot of it and adds a delta back.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a long roll of carbon paper that gets passed down an assembly line. Every station gets a copy of what is currently on the paper (the 'read'), works on its copy, and then stamps a small correction back onto the original paper (the 'write'). The paper itself is never erased; it just accumulates more and more refinements as it moves through the line. In a transformer, that carbon paper is the residual stream. The stations are the attention and feed-forward sublayers, and 'read a copy' is what the LayerNorm does at the start of each sublayer. Eighty stations later, the paper carries every refinement the model wanted to make.

Key concepts

Transformer architecture Attention

Quick scan · 2–4 min

Everything important, quickly.

3 min: define the residual stream as a fixed-shape running tensor, decompose each sublayer into read (LN) + write (add), note that nothing is ever overwritten, and connect to why mech-interp treats it as a shared bus.

Side-by-side

Aspect	Convolutional net	Transformer (residual stream)
What flows between layers	A transformed activation (replaced each layer)	A running residual tensor (added to each block)
Shape between layers	Can change (channels, spatial dims)	Constant at `(batch, seq_len, d_model)`
What 'depth' means	Sequential transformation	Additive refinement on a shared bus
Information from layer 0	Often lost or compressed	Persists structurally to the top

Real-world examples

Real products, models, and research that use this idea.

Anthropic's 'A Mathematical Framework for Transformer Circuits' (2021) introduced the residual-stream-as-bus framing that became standard in mech-interp.
Llama 3.1 8B carries a residual stream of shape `(batch, seq_len, 4096)` through 32 blocks; the same tensor is the input to the final RMSNorm and unembedding head.
Activation patching experiments (used in interpretability papers like ROME and IOI) work by editing a single position in the residual stream at a chosen layer and observing downstream effects.
The 'logit lens' technique (nostalgebraist, 2020) projects the residual stream at every layer through the unembedding head to read out what the model is implicitly predicting at each depth.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf the residual stream is just `x + delta_1 + delta_2 + ...`, why do interpretability researchers care about which sublayer wrote which delta?

Sign in or go Pro to see the approach for this follow-up.

QWhat changes about the residual-stream picture under post-norm?

Sign in or go Pro to see the approach for this follow-up.

QWhy does widening d_model usually help quality more than adding extra attention heads?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.