Spot the error in this description of pre-norm placement and the final norm

**The text inverts two things.** First, it puts the norm in the wrong place. Pre-norm means each sublayer's INPUT is normalized; the residual addition then happens between the un-normalized stream and the sublayer output. Written out: $$y = x + \text{sublayer}(\text{norm}(x))$$ Second, it draws the wrong conclusion about the final norm. Because the residual stream itself is never normalized inside the block, by the time you reach the top of the stack the accumulated activations can have arbitrary scale. So pre-norm architectures append a single LayerNorm or RMSNorm right before the unembedding head. Llama, Mistral, DeepSeek, Qwen, and Gemma all do this; their model definitions all end with `self.norm = RMSNorm(d_model)` applied to the output of the last block. **Where the text would be roughly right.** Post-norm: `y = norm(x + sublayer(x))`. Here the norm IS on top of the residual add. In a pure post-norm stack the residual is normalized at every layer, so a final norm before logits is redundant. But the text describes pre-norm, not post-norm, so its second claim does not follow. The practical reason pre-norm became dominant is the identity gradient path through the un-normalized residual: gradients reach every layer cleanly. The single final norm is the small price you pay for that.

**The pre-norm equation.** A pre-norm block is `y = x + sublayer(norm(x))`. The norm sits inside the sublayer branch; the residual `+` operates on `x` directly. The crucial property is that the residual stream `x` is never touched by a norm inside any block; only the inputs to the sublayer functions are normalized. **Why this matters for the final norm.** Because the residual stream accumulates additive updates across L blocks without normalization, its variance grows roughly with depth. At the top of the stack, the un-normalized residual can have a scale that depends on the number of layers. The unembedding head expects activations in a predictable scale for softmax to work well, so pre-norm architectures append ONE final norm right before logits: ``` x = embedding(tokens) for block in blocks: x = block(x) # pre-norm internally x = final_norm(x) logits = x @ unembed ``` This final norm is a one-line addition at the top of the model. In Llama, it is the `self.norm = RMSNorm(d_model)` you see right before the LM head. It is not optional; without it, logits scale grows with depth and softmax behavior degrades. **Why the text is wrong on both counts.** Its description matches post-norm, where `y = norm(x + sublayer(x))` and the residual stream IS normalized at every block. In that case, no final norm is needed because the last block's norm already normalized the output. But conflating the two architectures hides exactly the insight that explains why frontier pre-norm models (Llama-class) include a final norm and earlier post-norm models (original 2017 Transformer, BERT) do not.