**The two placements** differ in where you put the LayerNorm operation relative to the residual connection.
**Post-norm** (the original Vaswani 2017 layout) puts the LayerNorm after the residual add: `x_out = LN(x_in + f(x_in))`. The residual stream gets normalized at every layer.
**Pre-norm** puts the LayerNorm inside the residual branch, applied to the input of the attention or FFN before the operation: `x_out = x_in + f(LN(x_in))`. The residual stream itself flows through untouched; only the input to the sub-layer is normalized.
**Why pre-norm wins.** Gradients flowing back through the residual path are not attenuated by the LN. The residual stream has a direct identity path from layer 80 back to layer 1 without any intervening normalization. Post-norm normalizes the entire residual stream at every layer, which compounds across depth and degrades the gradient signal in early layers.
The original transformer was famously hard to train deep without **learning-rate warmup**: you had to ramp the LR up from a tiny value over the first few thousand steps, or training diverged. This warmup was a kludge to compensate for the gradient instability of post-norm. GPT-2 (2019) switched to pre-norm and the warmup requirement relaxed dramatically. Every modern decoder-only LLM since (GPT-3, Llama, Mistral, Gemma, Claude, Qwen) uses pre-norm.
The wrong-option trap is option A: 'post-norm is what GPT-3, Llama, and Mistral use'. This is the inverse of the historical fact. Post-norm is what the original 2017 paper used; modern models all use pre-norm.
**Pre-norm** is the modern default because it gives clean gradient flow through the residual stream and lets very deep stacks (50-100+ layers) train without elaborate learning-rate warmup.
The key difference:
- **Post-norm**: `x_{l+1} = LN(x_l + f_l(x_l))`. The residual stream is normalized at every layer.
- **Pre-norm**: `x_{l+1} = x_l + f_l(LN(x_l))`. The residual stream flows unchanged through the layer; only the sub-layer's input is normalized.
In pre-norm, the residual stream from input to output is a sum of layer contributions:
$$x_L = x_0 + \sum_{l=0}^{L-1} f_l(\text{LN}(x_l))$$
The gradient with respect to early layers flows back through the identity path, untouched by LN. In post-norm, the same gradient passes through L LayerNorm operations on its way back, each of which can attenuate or distort the signal.
**Historical context.** The original Vaswani 2017 transformer used post-norm. It was famously hard to train without learning-rate warmup; you had to ramp LR over several thousand steps to prevent divergence. GPT-2 (2019) switched to pre-norm and the warmup requirement became much less critical. Every decoder-only LLM since (GPT-3, GPT-4, Llama family, Mistral family, Gemma, Qwen, DeepSeek, Claude family) uses pre-norm.
A few details to know:
- Pre-norm typically pairs with **RMSNorm** (a simpler LayerNorm variant) in modern models because RMSNorm is faster on GPU.
- Pre-norm models often need a final LayerNorm before the LM head because the residual stream's variance grows unboundedly through depth.
- Some research has revisited post-norm with careful initialization (DeepNorm) for stability, but mainstream production has stayed on pre-norm.
Option A is the wrong-direction trap; it inverts the historical fact. Options C and D are factually false.
Quick reference
TL;DR
Pre-norm puts LayerNorm inside the residual branch so the residual path is never normalized. Gradients flow cleanly back, deep stacks train without warmup, and every modern decoder LLM uses it.
Imagine a long telephone game played across 80 people. Post-norm is like having each person whisper to the next but a referee retunes the volume after every pair speaks. The retuning makes early whispers fade away because every retune slightly attenuates what came before. Pre-norm is like setting the volume once before each person speaks but leaving the original whisper untouched in a backchannel. The backchannel carries the original message clean across all 80 people; the retuning only affects what each person says into the channel. The second setup lets messages survive long chains, which is exactly what gradients need to do in deep networks.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Shallow transformers under 12 layers where both placements train fine The gradient-attenuation argument only bites at depth; for BERT-style 12-24 layer models, post-norm works adequately and pre-norm is not strictly required.
Research stacks specifically studying post-norm with DeepNorm or similar stability tricks DeepNorm has shown 1000-layer training with post-norm via careful initialization; if you are reproducing or extending that work, post-norm is the right choice.
Architectures with a different normalization scheme entirely (no LayerNorm at all) Some recent research explores normalization-free transformers using initialization tricks; the pre-norm vs post-norm question does not apply.
Models where you specifically want the residual stream's variance bounded at every layer Post-norm provides this bounding by normalizing after the add; pre-norm requires a final LayerNorm to compensate for unbounded variance growth.
Quick scan · 2–4 min
Everything important, quickly.
Compare the formulas, explain the gradient-flow argument for pre-norm's residual identity path, walk the historical transition from Vaswani 2017 to GPT-2 to modern LLMs, address the variance-growth concern and the final-LayerNorm fix, and close with DeepNorm and RMSNorm as related variants.
Side-by-side
Property
Post-norm
Pre-norm
Formula
x_out = LN(x_in + f(x_in))
x_out = x_in + f(LN(x_in))
Residual path through LN?
Yes (every layer)
No (identity path)
Gradient stability at depth
Poor without warmup
Stable with standard init
Used by GPT-1, BERT, Vaswani 2017
Yes
No
Used by GPT-2+, Llama, Mistral, modern LLMs
No
Yes
Needs final LN before LM head
No
Yes (variance grows)
Real-world examples
Real products, models, and research that use this idea.
1GPT-2 (2019) switched from post-norm (used in GPT-1 following Vaswani) to pre-norm and dramatically reduced warmup requirements.
2GPT-3/4/5 series all use pre-norm; Llama 1/2/3/4 Maverick all use pre-norm with RMSNorm.
3Mistral, Mixtral, Qwen 3.5, DeepSeek V4, Gemma 4, and Claude Opus 4.7 all use pre-norm with RMSNorm.
4DeepNorm (Wang et al. 2022) is a notable academic effort to stabilize post-norm at extreme depth (1000+ layers) via initialization, but it has not displaced pre-norm in mainstream production.
5BERT uses post-norm but is only 12-24 layers deep, where the difference is less pronounced.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does pre-norm's residual-stream variance grow unboundedly with depth, and what fixes it?
Sign in or go Pro to see the approach for this follow-up.
QHow does DeepNorm (Wang et al. 2022) make post-norm work at extreme depth?
Sign in or go Pro to see the approach for this follow-up.
QWhy have modern LLMs converged on pre-norm + RMSNorm rather than pre-norm + LayerNorm?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Believing the original Vaswani layout (post-norm) is what modern LLMs use. GPT-2 already switched to pre-norm in 2019 and every frontier LLM since has followed.