Explain mechanically why pre-norm trains stably out to 100+ layers without warmup, while post-norm requires a careful learning rate schedule and tends to destabilize past about 20 layers.
Free · 2 AI evals / day
**Pre-norm and post-norm differ in one structural way: whether the normalization happens BEFORE the sublayer (pre-norm) or AFTER the residual add (post-norm).** That single difference changes the entire training-stability story.
**Pre-norm structure.** Each block computes `x = x + Sublayer(LN(x))`. The residual stream `x` flows through every block as a pure identity, with each sublayer adding a normalized correction. Backprop sees a chain of identity plus delta updates: the gradient with respect to the input is roughly the gradient with respect to the output, plus a small contribution from the sublayer. Through 100 blocks, this stays well-behaved because adding small corrections does not compound the way multiplying does.
**Post-norm structure.** Each block computes `x = LN(x + Sublayer(x))`. The residual stream itself gets rescaled at every block by the LayerNorm. Through depth, the residual scale and the gradient norm both drift unpredictably. The optimizer has to use a careful learning rate warmup schedule (linear ramp over thousands of steps from near-zero up to the peak rate) plus controlled initialization to keep both stable.
**The empirical consequence.** Post-norm models tend to destabilize past about 20 layers without warmup tricks. Pre-norm trains reliably past 100 layers with no special handling. Every modern LLM since GPT-3 (Llama, Mistral, Qwen, DeepSeek, Claude-family) uses pre-norm. The 2017 Vaswani paper used post-norm with 6-layer encoder and decoder, which was shallow enough that the instability did not bite.
**The mechanical difference comes down to where the LayerNorm sits relative to the residual add, and that changes whether the residual stream remains an identity path.**
**Pre-norm: identity residual.** The block computation is:
```
x = x + Sublayer(LN(x))
```
The LayerNorm runs on the sublayer's INPUT, the residual stream `x` is added back at the end without any rescaling. The residual path from block to block is the identity function. Through 100 blocks, the residual stream looks like:
```
x_100 = x_0 + Sublayer_0(LN(x_0)) + Sublayer_1(LN(x_1)) + ... + Sublayer_99(LN(x_99))
```
It is a SUM of normalized sublayer contributions added to the original input. Backprop through this sum is well-behaved: the gradient with respect to `x_0` is the gradient with respect to `x_100` plus a sum of sublayer gradients. The chain rule gives `dL/dx_0 ≈ dL/dx_100 * (I + small)`, which stays close to `dL/dx_100` even through arbitrary depth. No vanishing, no explosion.
**Post-norm: rescaled residual.** The block computation is:
```
x = LN(x + Sublayer(x))
```
The LayerNorm runs AFTER the residual add. The residual stream itself is rescaled at every block. Through depth, the chain rule gives `dL/dx_0 = dL/dx_100 * prod_i (dLN_i / dx_i)`, where each `dLN_i / dx_i` is a Jacobian whose effective scale depends on the input distribution to that layer. Through 100 blocks, this product compounds: small changes in any one layer's normalization scale propagate multiplicatively, and gradients tend to either vanish or explode.
**Why warmup compensates.** Linear learning rate warmup over the first 4000-10000 steps starts with a near zero learning rate, lets the model find a stable parameter regime where the per-layer LayerNorm scales are roughly matched, then ramps the learning rate up to the peak. This works because the destructive multiplicative dynamics are worst when individual layer parameters are poorly initialized.
Quick reference
TL;DR
Pre-norm keeps the residual stream as a pure identity path, so gradients flow through depth as a sum (not a product) of sublayer contributions.
Picture the residual stream as a highway running straight through the model from input to output. In pre-norm, every block is an exit ramp that adds a small correction back onto the highway, then merges back in. The highway itself never gets rescaled, it just keeps going. Gradients flowing backward have a clean, unobstructed lane to travel down, so even with 100 blocks the signal stays strong. In post-norm, every block puts up a toll booth on the highway itself, rescaling the whole stream after the merge. With 24 such toll booths the stream's scale starts drifting unpredictably and the model gets very hard to train. To make post-norm work past about 20 layers, engineers have to add learning-rate warmup that slowly lets the network find good early weights before the toll booths start interacting badly. Pre-norm avoids the toll booths altogether, which is why every modern LLM with more than a dozen layers uses it.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Training a shallow (6-12 layer) transformer like BERT-base for a specific task At that depth post-norm trains fine and sometimes gives marginally better final performance because the residual rescaling acts as implicit regularization. The pre-norm advantage only matters past about 20 layers.
Reproducing a historical paper (e.g., the original 2017 Vaswani transformer) The 2017 paper used post-norm with 6-layer encoder/decoder. Switching to pre-norm would not reproduce the published numbers and would defeat the purpose of historical reproduction.
Researching novel normalization placements (DeepNorm, ADMIN) Those works deliberately study post-norm because the goal is to FIX it via initialization rather than sidestep it via pre-norm. Switching to pre-norm would skip the research question entirely.
Quick scan · 2–4 min
Everything important, quickly.
5 min: state the structural difference, give the residual decomposition for each, explain the sum-vs-product gradient flow, mention the depth cliff and warmup, close with why modern LLMs all pick pre-norm.
Real-world examples
Real products, models, and research that use this idea.
1Llama 3.1 70B: 80 layers, pre-norm with RMSNorm. Trains reliably without exotic warmup. Would be impossible at this depth with post-norm.
2GPT-3 (175B, 96 layers): the model that established pre-norm as the universal default. Pre-norm was a key reason GPT-3 trained without the instability that plagued earlier scale-up attempts.
3BERT-base (12 layers, post-norm): one of the last successful post-norm production models. At 12 layers post-norm is fine and was the natural choice in 2018.
4DeepNorm paper (Wang et al., 2022): trained a post-norm transformer to 1000 layers using a custom initialization. Demonstrates that post-norm is stabilizable in principle, but the engineering effort is significant.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the residual stream's scale a problem in post-norm specifically?
Sign in or go Pro to see the approach for this follow-up.
QCould you build a post-norm transformer that is stable at 100 layers?
Sign in or go Pro to see the approach for this follow-up.
QWhy does pre-norm sometimes underperform post-norm at shallow depth?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Believing pre-norm vs post-norm is just about where the LayerNorm symbol appears in the code. It is actually about whether the residual stream stays an identity path or gets rescaled at every block, which changes the gradient-norm behavior through depth.