Diagnose why a 48-layer model diverges right after a pre-norm to post-norm refactor

Short answer·Medium·4.0 · 0·~3 min·Asked atBcg Notion Tencent

Attempt it

To reproduce an older paper, you refactor a 48-layer decoder from pre-norm to post-norm, keep the identical learning rate schedule with no warmup, and relaunch. Loss diverges within 300 steps; the pre-norm run was stable. Diagnose the root cause and propose two concrete fixes that keep post-norm.

Free · 2 AI evals / day

**The architectures differ at one line.** Pre-norm computes `x = x + sublayer(norm(x))`. Post-norm computes `x = norm(x + sublayer(x))`. The norm moved from before the sublayer to after the residual add. That one-line move changes the gradient story dramatically. In pre-norm, the residual `+` provides an identity path from the loss back to every layer. Gradients flow back cleanly without depth-dependent amplification. In post-norm, the norm sits on top of the residual add, so the gradient through it depends on the activation scale at that layer, and the cumulative effect across 48 layers makes gradients at the bottom either too big or too small at initialization. The first few optimizer steps with a cold model and no warmup take the loss off a cliff. Your pre-norm run was tolerant because the identity path kept gradients sane regardless of LR; your post-norm run is intolerant because the math says so. **Fix 1: add warmup.** The 2017 Transformer was post-norm and shipped with `lr = d_model^-0.5 * min(step^-0.5, step * warmup^-1.5)` precisely because cold starts at full LR diverge. A linear warmup over 4000-10000 steps is the standard modern equivalent. Optionally lower the peak LR too.

**The mechanism.** Pre-norm keeps an identity path from loss to every layer because the residual `x + sublayer(norm(x))` passes gradients through the `+` unchanged. Post-norm's `norm(x + sublayer(x))` puts the norm on top of the add, so the gradient through it is scaled by the LayerNorm Jacobian at every layer. Stack 48 of these and the gradient at layer 1 picks up a depth-dependent scale factor that is sensitive to the activation magnitudes early in training. **Why the first 300 steps blow up.** At init, sublayer outputs have variance proportional to their projections; with standard Xavier or Kaiming init, the residual stream's variance grows roughly linearly with depth in post-norm. The output norm at the top then scales loss gradients by a factor that, after backpropagating through 48 layers, produces parameter updates large enough to overshoot. The first optimizer step lands in a region where the LayerNorm denominators are different from what the rest of the network expects, and the loss explodes. **Pre-norm tolerated the same schedule** because its identity gradient path makes the effective depth multiplicative factor close to 1 regardless of layer count. The original GPT-2 codebase showed you can train 48+ pre-norm layers with minimal warmup. **Fix 1: warmup.** Add 4000-10000 steps of linear LR warmup. This is what the 2017 paper's inverse-sqrt schedule does implicitly. Optionally lower the peak LR by 2-4x if you want extra margin.

TL;DR

Post-norm breaks the identity gradient path that pre-norm provides, so early-training gradients are depth-amplified and the first cold-start updates blow up. Fix with LR warmup and depth-aware residual scaling.

Memory aid

Easy to grasp

Pre-norm is like a building with a fire-pole going from every floor straight to the ground. Information can fall through cleanly without being squeezed. Post-norm replaces each pole section with a revolving door that scales whatever passes through. Run cold and a single big push at the top spins all 48 doors and the energy at the bottom is huge. With a slow ramp-up (warmup), the doors get oiled gradually; with smaller doors at each floor (initialization scaling), the energy stays bounded. The original 2017 Transformer was post-norm and shipped with exactly this kind of slow ramp-up because the team had figured the problem out.

Key concepts

Transformer architecture

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Post-norm vs pre-norm is the cleanest example in transformer architecture of how a one-line equation change reshapes the training recipe. The interview-friendly version of this answer covers the gradient mechanism, the historical fix (warmup), the modern fix (depth-aware init), and why pre-norm became the 2026 default.

The failure mode in the prompt is doing exactly what the math says it should: stripping the identity gradient path and running a cold-start optimizer on a 48-layer stack is mechanically destabilizing. Treating this as a code bug wastes hours; recognizing the architectural cause solves it in one PR.

The two block equations side by side

Pre-norm. y = x + sublayer(LN(x)). The LayerNorm sits inside the sublayer branch; the residual stream x passes through the outer + un-normalized.

Post-norm. y = LN(x + sublayer(x)). The sublayer reads the un-normalized residual stream; the LayerNorm sits on top of the residual add.

The gradient through pre-norm is dy/dx = I + (dsublayer/dLN) * (dLN/dx). The I term is the identity path: any gradient from the loss reaches x with at least a multiplicative factor of 1 unchanged.

The gradient through post-norm is dy/dx = (dLN/d_input) * (I + dsublayer/dx). The LayerNorm Jacobian dLN/d_input multiplies everything, including the identity term. Across L layers, the product of L LayerNorm Jacobians depends on the activation scale at every layer and is sensitive to initialization.

This structural asymmetry is everything. Pre-norm gives a clean signal path; post-norm does not.

What goes wrong in the first 300 steps

Fix 1: learning-rate warmup

Fix 2: depth-aware residual scaling

Why pre-norm became the 2026 default

Putting numbers to it: 2026 frontier-model context

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

The original Transformer (Vaswani et al., 2017) was post-norm and shipped with the inverse-sqrt warmup schedule precisely to keep cold starts stable.
GPT-2 (OpenAI, 2019) switched to pre-norm and demonstrated stable 48-layer training with minimal warmup; this drove the open-source convention.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does DeepNet's residual scaling differ from `1/sqrt(2L)` output projection scaling?

DeepNet scales the residual branch input by a layer-dependent beta and the initialization of the sublayer weights by alpha, with both factors derived to bound Lipschitz constants. 1/sqrt(2L) is a simpler one-knob version that addresses output variance only.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Blaming the divergence on a code bug or bad data. The architecture change itself made the existing schedule incompatible; the run is doing exactly what the math predicts.

60 second bullets to scan on the way to the call.

Write the pre-norm and post-norm block equations side by side and identify where the norm sits
Explain the identity gradient path in pre-norm

Primary sources. Browse if you want the original framing.