Pre-norm versus post-norm: which placement makes deep stacks stable?

**The two placements** differ in where you put the LayerNorm operation relative to the residual connection. **Post-norm** (the original Vaswani 2017 layout) puts the LayerNorm after the residual add: `x_out = LN(x_in + f(x_in))`. The residual stream gets normalized at every layer. **Pre-norm** puts the LayerNorm inside the residual branch, applied to the input of the attention or FFN before the operation: `x_out = x_in + f(LN(x_in))`. The residual stream itself flows through untouched; only the input to the sub-layer is normalized. **Why pre-norm wins.** Gradients flowing back through the residual path are not attenuated by the LN. The residual stream has a direct identity path from layer 80 back to layer 1 without any intervening normalization. Post-norm normalizes the entire residual stream at every layer, which compounds across depth and degrades the gradient signal in early layers. The original transformer was famously hard to train deep without **learning-rate warmup**: you had to ramp the LR up from a tiny value over the first few thousand steps, or training diverged. This warmup was a kludge to compensate for the gradient instability of post-norm. GPT-2 (2019) switched to pre-norm and the warmup requirement relaxed dramatically. Every modern decoder-only LLM since (GPT-3, Llama, Mistral, Gemma, Claude, Qwen) uses pre-norm. The wrong-option trap is option A: 'post-norm is what GPT-3, Llama, and Mistral use'. This is the inverse of the historical fact. Post-norm is what the original 2017 paper used; modern models all use pre-norm.

**Pre-norm** is the modern default because it gives clean gradient flow through the residual stream and lets very deep stacks (50-100+ layers) train without elaborate learning-rate warmup. The key difference: - **Post-norm**: `x_{l+1} = LN(x_l + f_l(x_l))`. The residual stream is normalized at every layer. - **Pre-norm**: `x_{l+1} = x_l + f_l(LN(x_l))`. The residual stream flows unchanged through the layer; only the sub-layer's input is normalized. In pre-norm, the residual stream from input to output is a sum of layer contributions: $$x_L = x_0 + \sum_{l=0}^{L-1} f_l(\text{LN}(x_l))$$ The gradient with respect to early layers flows back through the identity path, untouched by LN. In post-norm, the same gradient passes through L LayerNorm operations on its way back, each of which can attenuate or distort the signal. **Historical context.** The original Vaswani 2017 transformer used post-norm. It was famously hard to train without learning-rate warmup; you had to ramp LR over several thousand steps to prevent divergence. GPT-2 (2019) switched to pre-norm and the warmup requirement became much less critical. Every decoder-only LLM since (GPT-3, GPT-4, Llama family, Mistral family, Gemma, Qwen, DeepSeek, Claude family) uses pre-norm. A few details to know: - Pre-norm typically pairs with **RMSNorm** (a simpler LayerNorm variant) in modern models because RMSNorm is faster on GPU. - Pre-norm models often need a final LayerNorm before the LM head because the residual stream's variance grows unboundedly through depth. - Some research has revisited post-norm with careful initialization (DeepNorm) for stability, but mainstream production has stayed on pre-norm. Option A is the wrong-direction trap; it inverts the historical fact. Options C and D are factually false.