Inside a transformer, what is the residual stream and how does attention interact with it?

Flashcard·Easy·4.0 · 0·~30s

Attempt it

The residual stream is the per-token d_model-wide vector that flows from block to block, top to bottom of the network. Each transformer block reads it, computes a delta, and adds that delta back. Inside an attention sub-layer the residual stream is read by the Q, K, and V projections, they are linear maps applied to the (normalized) residual stream vector at each token position. The attention output is then projected by W_O and ADDED back to the residual stream via a skip connection. The stream is what makes early-layer information accessible to late layers and what carries the model's per-token state.

Your answer

Quick reference

TL;DR

The residual stream is the d_model-wide per-token vector that flows untouched through the network; every sub-layer reads it via projections and writes a delta back via a skip add.

Memory aid

Easy to grasp

Picture a long conveyor belt running through a factory. At every station, a worker reaches over, looks at what is on the belt, builds a small modification, and drops the modification back onto the belt without removing the original. By the time the conveyor reaches the end, the product carries the original input plus every worker's contribution layered on top. That conveyor belt is the residual stream. Attention is one type of worker; the feed-forward layer is another. They all read the belt and add to it, never replacing it.

Key concepts

Residual stream Residual connection

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Define the stream as the persistent d_model-wide per-token vector, walk how attention reads via projections and writes via the skip add, contrast pre-norm vs post-norm preservation of the stream, connect to interpretability.

Real-world examples

Real products, models, and research that use this idea.

Anthropic's transformer-circuits work models induction heads as residual-stream circuits where a head in one layer writes to subspaces a head in a later layer reads.
Llama 4 Maverick maintains an 8192-wide residual stream across 80+ blocks; sub-layer contributions accumulate before the final projection.
Linear probing in interpretability research treats the residual stream at each layer as a feature vector that can be probed for syntactic and semantic properties.
Mistral Large 2 uses pre-norm RMSNorm to preserve the residual stream as an unnormalized signal bus throughout the network.
ROME and MEMIT model editing techniques work by modifying FFN W_out matrices, which writes specific subspaces to the residual stream encoding a target fact.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you read out the contribution of a specific attention head to the residual stream at a given layer?

QWhy does d_model need to scale with model depth to maintain quality?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Thinking each layer REPLACES the activations rather than ADDS to them. The whole architectural point is that information persists down the stack via the unmodified skip path.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.