RMSNorm versus LayerNorm, what is kept and what is dropped?

MCQ·Medium·4.0 · 0·~1 min

Attempt it

**LayerNorm** does three things to an activation vector x: 1. **Center**: subtract the per-token mean. 2. **Scale**: divide by the per-token standard deviation. 3. **Affine**: multiply by a learned vector gamma and add a learned vector beta. $$\text{LN}(x) = \gamma \cdot \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$ **RMSNorm** keeps only steps 2 and the multiplicative part of step 3. It drops: - The mean centering (no `x - mu`). - The additive bias (no `+ beta`). What remains: $$\text{RMSNorm}(x) = \gamma \cdot \frac{x}{\sqrt{\text{mean}(x^2) + \epsilon}}$$ The normalization step is now 'divide by the root mean square of x' (hence the name) instead of 'divide by the standard deviation'. Without the mean subtraction, RMS is the right scale measure. ### Why drop centering and bias? Zhang and Sennrich 2019 found empirically that for transformer language models, the mean centering and the additive bias contribute negligibly to quality. The divide by scale step is what stabilizes activations; the other operations are mostly redundant. ### What you gain Fewer FLOPs per normalization call (no mean computation, no bias add). Fewer parameters (no beta). Slightly more stable behavior under low-precision (bf16, fp8) training because there is one less reduction operation. At 70B parameters and 128k context, those small per-call savings add up across hundreds of thousands of calls per forward pass.

**LayerNorm** centers and scales an activation vector then applies a learned affine: `LN(x) = gamma * (x - mu) / sqrt(sigma^2 + eps) + beta`. **RMSNorm** drops the mean centering and the additive bias, keeping only the divide-by-RMS step and the learnable scale: `RMSNorm(x) = gamma * x / sqrt(mean(x^2) + eps)`. The canonical operation: $$\text{RMSNorm}(x) = \gamma \cdot \frac{x}{\sqrt{\text{mean}(x^2) + \epsilon}}$$ What is KEPT: the variance-style normalization (now in RMS form rather than std form, since without mean subtraction, RMS is the natural scale) and the per-element learnable scale gamma. What is DROPPED: the per-token mean subtraction (`x - mu` becomes just `x`) and the additive bias `beta`. Zhang and Sennrich 2019 introduced RMSNorm and showed empirically that for transformer LMs the dropped operations contribute negligibly to quality. The intuition: the scale of activations matters for softmax stability and gradient flow; the centering and bias add representational capacity but the rest of the network (W_Q, W_K, W_V projections; FFN biases) can absorb the equivalent expressiveness. ### Cost savings at scale - Fewer FLOPs per call (no mean computation, no bias add). - Fewer parameters (no `beta` of size d_model per norm). - Better numerical stability under low-precision training (one less reduction operation that can underflow in fp8/bf16). - Cleaner CUDA kernel (less memory bandwidth for the reduction step). ### Adoption Llama (all generations), Mistral, Gemma, Qwen, DeepSeek, all use RMSNorm. The original 2017 transformer and BERT use LayerNorm. The migration started around 2023 with Llama 1 and is now near-universal for new LLMs.

Quick reference

TL;DR

RMSNorm keeps the divide-by-RMS step and the learnable scale, drops the mean centering and the additive bias. Cheaper, fewer parameters, no measurable quality loss.

Memory aid

Easy to grasp

Picture LayerNorm as a quality-control station that does three jobs: center the product on the conveyor belt (subtract the mean), shrink or stretch it to a standard size (divide by standard deviation), then apply a known label (multiply by gamma plus add beta). RMSNorm decides two of those jobs are unnecessary overhead: it skips the centering and the label-adding, keeping only the resize step plus a multiplier. Same product comes out the other side at roughly the same quality, with fewer operations per item. That is why every modern open-weight LLM swapped LayerNorm for RMSNorm.

Key concepts

LayerNorm RMSNorm Normalization layers

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Write both formulas, name what is dropped (mean centering, additive bias), justify why this is safe (downstream projections absorb the redundancy), list the savings, give 2026 LLM examples.

Side-by-side

Property	LayerNorm	RMSNorm
Mean centering	Yes (x - mu)	No
Scale normalization	Divide by std	Divide by RMS
Learned scale gamma	Yes	Yes
Additive bias beta	Yes	No
Reductions per call	2 (mean, var)	1 (mean of squares)
Parameters per norm	2 * d_model	1 * d_model
Modern LLM adoption	BERT, GPT-2, GPT-3	Llama 1-4, Mistral, Gemma, Qwen, DeepSeek

Real-world examples

Real products, models, and research that use this idea.

Llama 4 Maverick uses RMSNorm pre-norm throughout, gamma per d_model dimension, no bias.
Mistral Large 2 and Mistral Small use RMSNorm consistent with the Llama family convention.
Gemma 2 / Gemma 3 use RMSNorm with a slight variation (a 1 + gamma parameterization for stability).
DeepSeek V4 uses RMSNorm wrapping MLA-style attention sub-layers across all blocks.
The original 2017 Vaswani transformer and BERT used LayerNorm; the historical lineage for LayerNorm in transformers stops with GPT-3.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does retrofitting RMSNorm into a LayerNorm-trained model break the model?

QAre there cases where mean centering still matters for transformer training?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming RMSNorm and LayerNorm are equivalent or just a rename. They have different gradients and remove two specific operations from LayerNorm.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.