Two things wrap attention in every transformer block: a LayerNorm (or RMSNorm) and a residual connection. The standard pre-norm pattern is x + Attention(LN(x)), where LN is applied to the input first, then attention runs, then the result is added back to the original x. The residual lets gradients flow cleanly through dozens of layers, without it, deep transformers do not train. Modern decoder-only LLMs like Llama and Mistral use pre-norm with RMSNorm; the original 2017 transformer used post-norm with LayerNorm.
Every transformer block has a standard wrapper around the attention sub-layer. Two pieces:
1. **A normalization layer** (LayerNorm in the original 2017 transformer; RMSNorm in modern Llama, Mistral, Gemma). It standardizes the per-token activation vector to a known scale before attention sees it.
2. **A residual connection**, the skip path that adds the original input back to the attention output: `x + Attention(LN(x))`.
The modern pattern is **pre-norm**: normalize first, then run attention, then add to the residual. The original 2017 transformer used **post-norm**: run attention first, add residual, then normalize. Pre-norm is more stable for very deep stacks and is the default in every modern LLM.
The residual is the load-bearing piece. Without it, gradients have to flow through every attention sub-layer in series, and they degrade across the stack. Networks beyond a handful of layers stop training. With the residual, the gradient can take the skip path directly back to early layers, which is what makes 70- to 100-layer transformers trainable at all.
The normalization keeps activations in a stable range so attention's softmax does not saturate, which is the other half of why the wrapper is universal.
Two wrappers, always: a normalization and a residual connection.
The modern pre-norm pattern is `output = x + Attention(LN(x))`. The input x goes through normalization (LayerNorm or RMSNorm), then attention, then the attention output is added back to the original x via the skip connection. The same wrapper pattern repeats for the FFN sub-layer: `output = x + FFN(LN(x))`.
The normalization (`LN`) keeps per-token activation vectors at a stable scale before attention reads them. Without it, dot-product magnitudes drift, softmax saturates, gradients vanish on non-peak keys. LayerNorm centers and scales; RMSNorm drops the centering and the bias, keeping only the divide-by-RMS plus a learnable scale. Llama and Mistral use RMSNorm because it is cheaper with no measurable quality loss.
The residual connection is the architectural reason transformers train at depth. Without `x + ...`, the only path for the gradient is back through every attention block in series, and deep transformers do not train. With the residual, the gradient has a clean identity-path back to earlier layers (essentially a constant-1 derivative through the skip), so 70-, 80-, 96-layer stacks are stable.
Pre-norm vs post-norm matters: the original 2017 transformer used post-norm (`LN(x + Attention(x))`), which works but requires careful learning-rate warmup. Pre-norm is the modern default precisely because it allows training without warmup and scales better.
Quick reference
TL;DR
Every transformer block wraps attention in two things: a normalization (LayerNorm or RMSNorm) on the input and a residual skip that adds the attention output back to the original input.
Picture a really long assembly line where each station does something to a product, but the product is also carried past every station on a conveyor belt. Each station gets to see the product, do its little adjustment, and then add that adjustment to whatever was on the belt. If the station messes up, the original product still arrives at the end. That conveyor belt is the residual stream. And before any station touches the product, an inspector standardizes its size so the station gets predictable input, that is the normalization. Both wrappers exist because without them deep stacks of stations would either break the product completely or fail to train at all.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Very shallow architectures (1-2 layers) The residual gradient argument applies most strongly to deep stacks. A 2-layer network trains fine without residuals, and very shallow encoders sometimes drop residuals for parameter count or interpretability reasons.
Models with explicit gating instead of residuals (e.g., GRU-style gates) Gated mixing (e.g., `gate * F(x) + (1 - gate) * x`) is a generalization of residual addition with a learned mixing weight. Some architectures (Highway Networks, GLU variants) replace the unconditional add with gated mixing. The wrapper concept persists but the form differs.
When implementing attention as a kernel benchmark in isolation Studying attention's compute or memory cost in isolation is fine without the wrappers; they are architectural, not algorithmic. But you cannot train a full LM on bare attention without them.
When porting attention to non-transformer architectures (e.g., as an addition to a CNN) If attention is being grafted onto an existing architecture that has its own normalization and skip-connection scheme, double-wrapping can hurt. Match the host architecture's conventions.
Quick scan · 2–4 min
Everything important, quickly.
Name the two wrappers, distinguish pre-norm and post-norm, explain why the residual is load-bearing for depth, note LayerNorm vs RMSNorm, give modern LLM examples.
Real-world examples
Real products, models, and research that use this idea.
1Llama 4 Maverick uses pre-norm RMSNorm wrapping attention and FFN sub-layers across all 80+ blocks.
2Mistral Large 2 ships pre-norm RMSNorm with sliding-window attention inside the wrapper.
3GPT-2, GPT-3, GPT-4 all use pre-norm with LayerNorm in the original implementations.
4The original 2017 Vaswani transformer used post-norm LayerNorm, which required learning-rate warmup to train stably.
What an interviewer would ask next. Try answering before peeking at the approach.
QWalk through how the gradient flows through a residual connection and explain why this prevents vanishing gradients.
Sign in or go Pro to see the approach for this follow-up.
QWhy does pre-norm allow training without LR warmup while post-norm needs it?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Forgetting the residual or treating it as 'just nice to have'. Without the residual, gradients vanish across stacked layers and the network does not train.