After multi-head attention concatenates per-head outputs and passes them through W_O, the result is added back into the **residual stream**.
The full formula for the attention sub-block in a pre-norm transformer is:
$$x_\text{out} = x_\text{in} + W_O \cdot \text{Concat}(\text{head}_1, \ldots, \text{head}_h)$$
The addition matters. Without it, each layer would replace the previous layer's output, and information from early layers would be lost by the time the model is 10 or 20 layers deep. With the addition, each layer contributes a delta on top of what is already there, so the embedding signal from layer 0 is still readable at layer 80.
The addition also helps gradients. During training, gradients flow back through the skip path without passing through every attention operation, so they reach early layers with reasonable magnitude. Without skip connections, deep transformers do not train, the gradients vanish or explode.
This is why the wrong answers fail: replacement breaks the residual property, concatenation doubles the feature width every block (which the architecture is not designed for), and multiplicative gating is a different mechanism used by GLU and Mamba layers.
The attention sub-block writes to the **residual stream** by addition, not replacement.
The sequence inside a pre-norm transformer block is: apply LayerNorm, run multi-head attention, project the concatenated heads through W_O, then add the result to the unnormalized input:
$$x_\text{out} = x_\text{in} + W_O \cdot \text{Concat}(\text{head}_1, \ldots, \text{head}_h)$$
The addition is the defining design choice that makes deep transformer training tractable.
Two things flow from it. First, the residual stream serves as a shared bus that every layer reads from and writes to. Information from the embedding layer is preserved up through the final layer because nothing overwrites the stream, layers only add to it. Second, gradients during the backward pass have a direct path back to early layers through the skip connection, which means they do not vanish even at 80 or 100 layer depths.
Replacement would break both properties. Information would be lost layer by layer, and gradients would have to flow back through every attention computation, which is unstable at depth.
Concatenation would double the feature width on every block. After 10 blocks the model would be operating on a vector 1024 times wider than the embedding dimension. The architecture is not designed for this and it would explode compute and memory.
Multiplicative gating is the GLU and Mamba pattern. Standard transformer attention uses additive residuals. Both designs work but they are different commitments.
Quick reference
TL;DR
The attention block adds its W_O output back into the residual stream: x_out = x_in + W_O(concat(heads)). The skip connection keeps deep training stable.
Imagine a long whiteboard where every meeting in a building adds a few notes without erasing what was already there. Each meeting reads the current state, decides what to contribute, and writes those new notes alongside the old ones. The whiteboard never gets wiped. By the end of the day, the bottom of the board shows the original meeting and the top shows the latest, with everything in between visible. A transformer block is a meeting. The residual stream is the whiteboard. Attention writes its conclusions onto the stream by addition, never by overwrite, so signal from the first layer is still readable at the last layer.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Building a brand new architecture that intentionally uses multiplicative gating end to end, like a pure GLU stack Additive residuals and multiplicative gates have different inductive biases, mixing them inconsistently can produce unexpected training dynamics.
Shallow networks where vanishing gradients are not a concern Residual connections shine at depth; for 2 or 3 layer networks the engineering complexity may not be worth the benefit.
Architectures where every layer must transform the representation completely, like some encoder-decoder bridges Residual preservation can interfere with intended large representation shifts; explicit non-residual projections are sometimes used at cross-attention boundaries.
Quick scan · 2–4 min
Everything important, quickly.
6-8 min: residual addition formula + skip connection role + why replacement and concatenation fail + pre-norm vs post-norm + residual stream as feature bus + connection to interpretability.
Real-world examples
Real products, models, and research that use this idea.
1Every modern frontier transformer (Llama 4 Maverick, GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, Mistral, DeepSeek V4) uses pre-norm with residual additions inherited from the original 2017 transformer block.
2ResNet (2015) introduced the residual connection in vision and is the direct architectural ancestor.
3Anthropic's interpretability research on 'residual stream as a bus of features' depends on the additive property: features live as additive directions, not multiplicative gates.
4Mamba and other state-space models use a different mixing scheme, but even there the macro block structure preserves residual skip connections.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat changes if you switch from post-norm to pre-norm?
Sign in or go Pro to see the approach for this follow-up.
QWhy is identity initialization useful for residual blocks?
Sign in or go Pro to see the approach for this follow-up.
QHow does the residual stream connect to interpretability?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Thinking attention replaces the residual stream. It does not. The output is added to the residual, that addition is what makes deep transformers trainable.