Walk through why pre-norm trains stably past 100 layers while post-norm needs warmup tricks

Short answer·Medium·4.0 · 0·~3 min·Relevant atAnthropic Meta Mistral AI

Attempt it

Explain mechanically why pre-norm trains stably out to 100+ layers without warmup, while post-norm requires a careful learning rate schedule and tends to destabilize past about 20 layers.

Free · 2 AI evals / day

**Pre-norm and post-norm differ in one structural way: whether the normalization happens BEFORE the sublayer (pre-norm) or AFTER the residual add (post-norm).** That single difference changes the entire training-stability story. **Pre-norm structure.** Each block computes `x = x + Sublayer(LN(x))`. The residual stream `x` flows through every block as a pure identity, with each sublayer adding a normalized correction. Backprop sees a chain of identity plus delta updates: the gradient with respect to the input is roughly the gradient with respect to the output, plus a small contribution from the sublayer. Through 100 blocks, this stays well-behaved because adding small corrections does not compound the way multiplying does. **Post-norm structure.** Each block computes `x = LN(x + Sublayer(x))`. The residual stream itself gets rescaled at every block by the LayerNorm. Through depth, the residual scale and the gradient norm both drift unpredictably. The optimizer has to use a careful learning rate warmup schedule (linear ramp over thousands of steps from near-zero up to the peak rate) plus controlled initialization to keep both stable. **The empirical consequence.** Post-norm models tend to destabilize past about 20 layers without warmup tricks. Pre-norm trains reliably past 100 layers with no special handling. Every modern LLM since GPT-3 (Llama, Mistral, Qwen, DeepSeek, Claude-family) uses pre-norm. The 2017 Vaswani paper used post-norm with 6-layer encoder and decoder, which was shallow enough that the instability did not bite.

**The mechanical difference comes down to where the LayerNorm sits relative to the residual add, and that changes whether the residual stream remains an identity path.** **Pre-norm: identity residual.** The block computation is: ``` x = x + Sublayer(LN(x)) ``` The LayerNorm runs on the sublayer's INPUT, the residual stream `x` is added back at the end without any rescaling. The residual path from block to block is the identity function. Through 100 blocks, the residual stream looks like: ``` x_100 = x_0 + Sublayer_0(LN(x_0)) + Sublayer_1(LN(x_1)) + ... + Sublayer_99(LN(x_99)) ``` It is a SUM of normalized sublayer contributions added to the original input. Backprop through this sum is well-behaved: the gradient with respect to `x_0` is the gradient with respect to `x_100` plus a sum of sublayer gradients. The chain rule gives `dL/dx_0 ≈ dL/dx_100 * (I + small)`, which stays close to `dL/dx_100` even through arbitrary depth. No vanishing, no explosion. **Post-norm: rescaled residual.** The block computation is: ``` x = LN(x + Sublayer(x)) ``` The LayerNorm runs AFTER the residual add. The residual stream itself is rescaled at every block. Through depth, the chain rule gives `dL/dx_0 = dL/dx_100 * prod_i (dLN_i / dx_i)`, where each `dLN_i / dx_i` is a Jacobian whose effective scale depends on the input distribution to that layer. Through 100 blocks, this product compounds: small changes in any one layer's normalization scale propagate multiplicatively, and gradients tend to either vanish or explode. **Why warmup compensates.** Linear learning rate warmup over the first 4000-10000 steps starts with a near zero learning rate, lets the model find a stable parameter regime where the per-layer LayerNorm scales are roughly matched, then ramps the learning rate up to the peak. This works because the destructive multiplicative dynamics are worst when individual layer parameters are poorly initialized.

Quick reference

TL;DR

Pre-norm keeps the residual stream as a pure identity path, so gradients flow through depth as a sum (not a product) of sublayer contributions.

Memory aid

Easy to grasp

Picture the residual stream as a highway running straight through the model from input to output. In pre-norm, every block is an exit ramp that adds a small correction back onto the highway, then merges back in. The highway itself never gets rescaled, it just keeps going. Gradients flowing backward have a clean, unobstructed lane to travel down, so even with 100 blocks the signal stays strong. In post-norm, every block puts up a toll booth on the highway itself, rescaling the whole stream after the merge. With 24 such toll booths the stream's scale starts drifting unpredictably and the model gets very hard to train. To make post-norm work past about 20 layers, engineers have to add learning-rate warmup that slowly lets the network find good early weights before the toll booths start interacting badly. Pre-norm avoids the toll booths altogether, which is why every modern LLM with more than a dozen layers uses it.

Key concepts

Layer normalization

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: state the structural difference, give the residual decomposition for each, explain the sum-vs-product gradient flow, mention the depth cliff and warmup, close with why modern LLMs all pick pre-norm.

Real-world examples

Real products, models, and research that use this idea.

Llama 3.1 70B: 80 layers, pre-norm with RMSNorm. Trains reliably without exotic warmup. Would be impossible at this depth with post-norm.
GPT-3 (175B, 96 layers): the model that established pre-norm as the universal default. Pre-norm was a key reason GPT-3 trained without the instability that plagued earlier scale-up attempts.
BERT-base (12 layers, post-norm): one of the last successful post-norm production models. At 12 layers post-norm is fine and was the natural choice in 2018.
DeepNorm paper (Wang et al., 2022): trained a post-norm transformer to 1000 layers using a custom initialization. Demonstrates that post-norm is stabilizable in principle, but the engineering effort is significant.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the residual stream's scale a problem in post-norm specifically?

QCould you build a post-norm transformer that is stable at 100 layers?

QWhy does pre-norm sometimes underperform post-norm at shallow depth?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing pre-norm vs post-norm is just about where the LayerNorm symbol appears in the code. It is actually about whether the residual stream stays an identity path or gets rescaled at every block, which changes the gradient-norm behavior through depth.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.