Pre-norm versus post-norm: which placement makes deep stacks stable?

MCQ·Medium·4.0 · 0·~1 min·Asked atBaidu Ola Salesforce·Relevant atMeta

Attempt it

**The two placements** differ in where you put the LayerNorm operation relative to the residual connection. **Post-norm** (the original Vaswani 2017 layout) puts the LayerNorm after the residual add: `x_out = LN(x_in + f(x_in))`. The residual stream gets normalized at every layer. **Pre-norm** puts the LayerNorm inside the residual branch, applied to the input of the attention or FFN before the operation: `x_out = x_in + f(LN(x_in))`. The residual stream itself flows through untouched; only the input to the sub-layer is normalized. **Why pre-norm wins.** Gradients flowing back through the residual path are not attenuated by the LN. The residual stream has a direct identity path from layer 80 back to layer 1 without any intervening normalization. Post-norm normalizes the entire residual stream at every layer, which compounds across depth and degrades the gradient signal in early layers. The original transformer was famously hard to train deep without **learning-rate warmup**: you had to ramp the LR up from a tiny value over the first few thousand steps, or training diverged. This warmup was a kludge to compensate for the gradient instability of post-norm. GPT-2 (2019) switched to pre-norm and the warmup requirement relaxed dramatically. Every modern decoder-only LLM since (GPT-3, Llama, Mistral, Gemma, Claude, Qwen) uses pre-norm. The wrong-option trap is option A: 'post-norm is what GPT-3, Llama, and Mistral use'. This is the inverse of the historical fact. Post-norm is what the original 2017 paper used; modern models all use pre-norm.

**Pre-norm** is the modern default because it gives clean gradient flow through the residual stream and lets very deep stacks (50-100+ layers) train without elaborate learning-rate warmup. The key difference: - **Post-norm**: `x_{l+1} = LN(x_l + f_l(x_l))`. The residual stream is normalized at every layer. - **Pre-norm**: `x_{l+1} = x_l + f_l(LN(x_l))`. The residual stream flows unchanged through the layer; only the sub-layer's input is normalized. In pre-norm, the residual stream from input to output is a sum of layer contributions: $$x_L = x_0 + \sum_{l=0}^{L-1} f_l(\text{LN}(x_l))$$ The gradient with respect to early layers flows back through the identity path, untouched by LN. In post-norm, the same gradient passes through L LayerNorm operations on its way back, each of which can attenuate or distort the signal. **Historical context.** The original Vaswani 2017 transformer used post-norm. It was famously hard to train without learning-rate warmup; you had to ramp LR over several thousand steps to prevent divergence. GPT-2 (2019) switched to pre-norm and the warmup requirement became much less critical. Every decoder-only LLM since (GPT-3, GPT-4, Llama family, Mistral family, Gemma, Qwen, DeepSeek, Claude family) uses pre-norm. A few details to know: - Pre-norm typically pairs with **RMSNorm** (a simpler LayerNorm variant) in modern models because RMSNorm is faster on GPU. - Pre-norm models often need a final LayerNorm before the LM head because the residual stream's variance grows unboundedly through depth. - Some research has revisited post-norm with careful initialization (DeepNorm) for stability, but mainstream production has stayed on pre-norm. Option A is the wrong-direction trap; it inverts the historical fact. Options C and D are factually false.

TL;DR

Pre-norm puts LayerNorm inside the residual branch so the residual path is never normalized. Gradients flow cleanly back, deep stacks train without warmup, and every modern decoder LLM uses it.

Memory aid

Easy to grasp

Imagine a long telephone game played across 80 people. Post-norm is like having each person whisper to the next but a referee retunes the volume after every pair speaks. The retuning makes early whispers fade away because every retune slightly attenuates what came before. Pre-norm is like setting the volume once before each person speaks but leaving the original whisper untouched in a backchannel. The backchannel carries the original message clean across all 80 people; the retuning only affects what each person says into the channel. The second setup lets messages survive long chains, which is exactly what gradients need to do in deep networks.

Key concepts

Pre-norm transformer Post-norm transformer Layer normalization Residual stream

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The placement of LayerNorm relative to the residual connection is one of the smallest architectural choices in transformers and one of the most consequential for training stability. The wrong placement (post-norm at depth) requires elaborate learning-rate warmup to even train. The right placement (pre-norm) lets you train 100-layer stacks with vanilla initialization and minimal warmup. The difference is gradient flow through the residual path.

This deep dive walks the gradient-flow argument that motivates pre-norm, traces the historical transition from the Vaswani 2017 post-norm layout to GPT-2's pre-norm switch, addresses the variance-growth concern that comes with pre-norm and the final-LayerNorm fix, and surveys the modern landscape of normalization choices in production LLMs.

Mental model: pre-norm gives the residual stream a clean identity path that gradients can flow through unattenuated. Post-norm puts a LayerNorm on every residual path, compounding the gradient distortion across depth.

The two formulas and what each does to the residual path

The residual connection is the structural innovation that made very deep networks trainable. The standard recursion is:

x_{l+1} = x_l + f_l(x_l)

where f_l is the sub-layer (attention or FFN). The residual path lets gradients flow back through depth without passing through the (potentially attenuating) f_l operation.

LayerNorm is the standard normalization in transformers. The question is where to insert it.

Post-norm formula

x_{l+1} = \text{LN}(x_l + f_l(x_l))

The LN is applied AFTER the residual add. The entire residual stream gets normalized at every layer.

Pre-norm formula

x_{l+1} = x_l + f_l(\text{LN}(x_l))

The LN is applied INSIDE the residual branch, on the input to f_l. The residual path itself is never normalized.

Why the difference matters for gradient flow

Unroll pre-norm over L layers:

x_L = x_0 + \sum_{l=0}^{L-1} f_l(\text{LN}(x_l))

The Jacobian dx_L / dx_0 has an identity term (from the unattenuated residual sum) plus contributions from each f_l. Gradients reach x_0 cleanly through the identity path.

Unroll post-norm. The recursion is:

x_{l+1} = \text{LN}(x_l + f_l(x_l))

The Jacobian dx_L / dx_0 involves L compositions of LN's Jacobian along the gradient path. Each LN has Jacobian rank-deficient (removes the mean direction) and scales by inverse variance, which can be much less than 1 if activations have large variance. Compounded over L = 50 or 100 layers, the gradient signal in early layers degrades sharply.

The historical transition

The variance-growth concern and the final LayerNorm

Modern variants and edge cases

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Property	Post-norm	Pre-norm
Formula	x_out = LN(x_in + f(x_in))	x_out = x_in + f(LN(x_in))
Residual path through LN?	Yes (every layer)	No (identity path)
Gradient stability at depth	Poor without warmup	Stable with standard init
Used by GPT-1, BERT, Vaswani 2017	Yes	No
Used by GPT-2+, Llama, Mistral, modern LLMs	No	Yes
Needs final LN before LM head	No	Yes (variance grows)

Real products, models, and research that use this idea.

GPT-2 (2019) switched from post-norm (used in GPT-1 following Vaswani) to pre-norm and dramatically reduced warmup requirements.
GPT-3/4/5 series all use pre-norm; Llama 1/2/3/4 Maverick all use pre-norm with RMSNorm.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does pre-norm's residual-stream variance grow unboundedly with depth, and what fixes it?

Each layer adds a contribution f_l(LN(x_l)) to the residual without rescaling. The variance of the sum grows as the variance of one term times L. Models compensate with a final LayerNorm before the LM head, and often with output-projection initialization scaled by 1/sqrt(L) so each layer's contribution to the running variance is bounded.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Believing the original Vaswani layout (post-norm) is what modern LLMs use. GPT-2 already switched to pre-norm in 2019 and every frontier LLM since has followed.

60 second bullets to scan on the way to the call.

Formula difference between pre-norm and post-norm
Why pre-norm gives cleaner gradient flow through the residual path

Primary sources. Browse if you want the original framing.