RMSNorm versus LayerNorm, what is kept and what is dropped?

MCQ·Medium·4.0 · 0·~1 min·Asked atCursor Observe Ai·Relevant atMeta

Attempt it

**LayerNorm** does three things to an activation vector x: 1. **Center**: subtract the per-token mean. 2. **Scale**: divide by the per-token standard deviation. 3. **Affine**: multiply by a learned vector gamma and add a learned vector beta. $$\text{LN}(x) = \gamma \cdot \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$ **RMSNorm** keeps only steps 2 and the multiplicative part of step 3. It drops: - The mean centering (no `x - mu`). - The additive bias (no `+ beta`). What remains: $$\text{RMSNorm}(x) = \gamma \cdot \frac{x}{\sqrt{\text{mean}(x^2) + \epsilon}}$$ The normalization step is now 'divide by the root mean square of x' (hence the name) instead of 'divide by the standard deviation'. Without the mean subtraction, RMS is the right scale measure. ### Why drop centering and bias? Zhang and Sennrich 2019 found empirically that for transformer language models, the mean centering and the additive bias contribute negligibly to quality. The divide by scale step is what stabilizes activations; the other operations are mostly redundant. ### What you gain Fewer FLOPs per normalization call (no mean computation, no bias add). Fewer parameters (no beta). Slightly more stable behavior under low-precision (bf16, fp8) training because there is one less reduction operation. At 70B parameters and 128k context, those small per-call savings add up across hundreds of thousands of calls per forward pass.

**LayerNorm** centers and scales an activation vector then applies a learned affine: `LN(x) = gamma * (x - mu) / sqrt(sigma^2 + eps) + beta`. **RMSNorm** drops the mean centering and the additive bias, keeping only the divide-by-RMS step and the learnable scale: `RMSNorm(x) = gamma * x / sqrt(mean(x^2) + eps)`. The canonical operation: $$\text{RMSNorm}(x) = \gamma \cdot \frac{x}{\sqrt{\text{mean}(x^2) + \epsilon}}$$ What is KEPT: the variance-style normalization (now in RMS form rather than std form, since without mean subtraction, RMS is the natural scale) and the per-element learnable scale gamma. What is DROPPED: the per-token mean subtraction (`x - mu` becomes just `x`) and the additive bias `beta`. Zhang and Sennrich 2019 introduced RMSNorm and showed empirically that for transformer LMs the dropped operations contribute negligibly to quality. The intuition: the scale of activations matters for softmax stability and gradient flow; the centering and bias add representational capacity but the rest of the network (W_Q, W_K, W_V projections; FFN biases) can absorb the equivalent expressiveness. ### Cost savings at scale - Fewer FLOPs per call (no mean computation, no bias add). - Fewer parameters (no `beta` of size d_model per norm). - Better numerical stability under low-precision training (one less reduction operation that can underflow in fp8/bf16). - Cleaner CUDA kernel (less memory bandwidth for the reduction step). ### Adoption Llama (all generations), Mistral, Gemma, Qwen, DeepSeek, all use RMSNorm. The original 2017 transformer and BERT use LayerNorm. The migration started around 2023 with Llama 1 and is now near-universal for new LLMs.

TL;DR

RMSNorm keeps the divide-by-RMS step and the learnable scale, drops the mean centering and the additive bias. Cheaper, fewer parameters, no measurable quality loss.

Memory aid

Easy to grasp

Picture LayerNorm as a quality-control station that does three jobs: center the product on the conveyor belt (subtract the mean), shrink or stretch it to a standard size (divide by standard deviation), then apply a known label (multiply by gamma plus add beta). RMSNorm decides two of those jobs are unnecessary overhead: it skips the centering and the label-adding, keeping only the resize step plus a multiplier. Same product comes out the other side at roughly the same quality, with fewer operations per item. That is why every modern open-weight LLM swapped LayerNorm for RMSNorm.

Key concepts

LayerNorm RMSNorm Normalization layers

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

RMSNorm versus LayerNorm is one of those architectural choices that looks tiny on paper, just dropping two operations, but quietly shapes the cost and stability profile of every modern transformer language model. Llama, Mistral, Gemma, Qwen, DeepSeek: every frontier open-weight family in 2026 uses RMSNorm. The original 2017 transformer and BERT used LayerNorm. The migration was complete by Llama 1 in early 2023 and has not reversed.

This deep dive walks both formulas in detail, explains the algebra of what gets kept and dropped, justifies why the dropped operations are empirically safe in transformer LMs, quantifies the FLOP and parameter savings at scale, covers the numerical-stability benefits under low-precision training, and closes with the architectural lineage of normalization techniques in deep learning.

By the end you should be able to write both formulas from memory, explain why the swap is safe at pretraining time but not at inference time, and name the 2026 models that use each.

The two formulas, side by side

Both LayerNorm and RMSNorm act per-token on a d_model-wide activation vector x.

LayerNorm

\text{LN}(x) = \gamma \cdot \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta

where:

mu = mean(x) over the d_model axis.
sigma^2 = mean((x - mu)^2) over the d_model axis.
gamma, beta are learnable vectors of shape (d_model,).
eps is a small constant for numerical stability (typically 1e-5).

Three distinct operations: center (subtract mean), scale (divide by std), affine (multiply gamma plus bias).

RMSNorm

\text{RMSNorm}(x) = \gamma \cdot \frac{x}{\sqrt{\frac{1}{d}\sum x_i^2 + \epsilon}}

where:

The denominator is the RMS of x: square each element, take the mean, square root.
gamma is a learnable vector of shape (d_model,).
eps is the same numerical stability constant.
NO beta.

Two distinct operations: scale by RMS, multiply by gamma.

The algebraic relationship

If mu = 0, then sigma = RMS (since variance = mean((x - mu)^2) = mean(x^2) when mu = 0). So in that special case, LayerNorm's normalization step collapses to RMSNorm's. RMSNorm can be thought of as 'LayerNorm under the implicit assumption that x has zero mean'.

Transformer activations are roughly mean-centered to begin with (the input embeddings are initialized that way; downstream projections preserve this approximately). So the assumption holds well enough in practice that the centering step provides little benefit.

Why the dropped operations are empirically safe

Cost savings at scale

Why retrofitting fails

The normalization-technique landscape

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Property	LayerNorm	RMSNorm
Mean centering	Yes (x - mu)	No
Scale normalization	Divide by std	Divide by RMS
Learned scale gamma	Yes	Yes
Additive bias beta	Yes	No
Reductions per call	2 (mean, var)	1 (mean of squares)
Parameters per norm	2 * d_model	1 * d_model
Modern LLM adoption	BERT, GPT-2, GPT-3	Llama 1-4, Mistral, Gemma, Qwen, DeepSeek

Real products, models, and research that use this idea.

Llama 4 Maverick uses RMSNorm pre-norm throughout, gamma per d_model dimension, no bias.
Mistral Large 2 and Mistral Small use RMSNorm consistent with the Llama family convention.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does retrofitting RMSNorm into a LayerNorm-trained model break the model?

The downstream projections (W_Q, W_K, W_V, FFN W_in) and biases have learned to expect mean-centered input from LayerNorm. Without centering, the projections see a shifted distribution, the bias terms encode incorrect offsets, and the resulting Q, K, V vectors fall outside the regime the attention softmax was trained for. Quality collapses immediately.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Claiming RMSNorm and LayerNorm are equivalent or just a rename. They have different gradients and remove two specific operations from LayerNorm.

60 second bullets to scan on the way to the call.

Write down both formulas: LayerNorm and RMSNorm
List the two operations RMSNorm drops

Primary sources. Browse if you want the original framing.