What does RMSNorm keep from LayerNorm, and what does it drop?

MCQ·Medium·4.0 · 0·~1 min·Asked atCursor Observe Ai·Relevant atMeta

Attempt it

**LayerNorm** does three things to an activation vector x: 1. **Center**: subtract the token's mean. 2. **Scale**: divide by the token's standard deviation. 3. **Affine**: multiply by a learned gamma and add a learned beta. $$\text{LN}(x) = \gamma \cdot \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$ **RMSNorm** keeps only steps 2 and the multiplicative part of step 3. It drops: - The mean centering (no `x - mu`). - The additive bias (no `+ beta`). What remains: $$\text{RMSNorm}(x) = \gamma \cdot \frac{x}{\sqrt{\text{mean}(x^2) + \epsilon}}$$ The normalization step is now 'divide by the root mean square of x' (hence the name) instead of 'divide by the standard deviation'. Without the mean subtraction, RMS is the right scale measure. ### Why drop centering and bias? Zhang and Sennrich 2019 found that for transformer language models, centering and bias contribute negligibly to quality. The divide by scale step stabilizes activations. ### What you gain You get fewer FLOPs (math operations) per call (no mean computation, no bias add) and fewer parameters (no beta). Low-precision (bf16, fp8) training is slightly more stable, with one fewer reduction. At 70B parameters and 128k context, those small savings add up across millions of per token norm calls.

**LayerNorm** centers and scales an activation vector then applies a learned affine: `LN(x) = gamma * (x - mu) / sqrt(sigma^2 + eps) + beta`. **RMSNorm** drops the mean centering and the additive bias, keeping only the divide-by-RMS step and the learnable scale: `RMSNorm(x) = gamma * x / sqrt(mean(x^2) + eps)`. The standard form: $$\text{RMSNorm}(x) = \gamma \cdot \frac{x}{\sqrt{\text{mean}(x^2) + \epsilon}}$$ What is KEPT: the variance-style normalization (now in RMS form rather than std form, since without mean subtraction, RMS is the natural scale) and the per-element learnable scale gamma. What is DROPPED: the per-token mean subtraction (`x - mu` becomes just `x`) and the additive bias `beta`. Zhang and Sennrich 2019 introduced RMSNorm and showed that for transformer LMs the dropped operations contribute negligibly to quality. The intuition: the scale of activations matters for softmax stability and gradient flow. The centering and bias add representational capacity, but the rest of the network (W_Q, W_K, W_V projections; FFN biases) can absorb the equivalent expressiveness. ### Cost savings at scale - Fewer FLOPs per call (no mean computation, no bias add). - Fewer parameters (no `beta` of size d_model per norm). - Better numerical stability under low-precision training (one fewer reduction). - Simpler CUDA kernel (less memory bandwidth). ### Adoption Llama (all generations), Mistral, Gemma, Qwen, and DeepSeek all use RMSNorm. The original 2017 transformer and BERT use LayerNorm. Llama 1 (2023) made RMSNorm the open-weight default (T5 used an RMS-style norm in 2019), and it is now near-universal.

TL;DR

RMSNorm keeps the divide-by-RMS step and the learnable scale. It drops the mean centering and the additive bias. It is cheaper, with fewer parameters and no measurable quality loss.

Memory aid

Easy to grasp

Picture LayerNorm as a quality-control station that does three jobs. It centers the product on the conveyor belt (subtract the mean). It shrinks or stretches it to a standard size (divide by standard deviation). Then it applies a known label (multiply by gamma plus add beta). RMSNorm decides two of those jobs are unnecessary overhead: it skips the centering and the label-adding, keeping only the resize step plus a multiplier. The same product comes out the other side at roughly the same quality, with fewer operations per item. That is why every modern open-weight LLM swapped LayerNorm for RMSNorm.

Key concepts

LayerNorm RMSNorm Normalization layers

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

RMSNorm versus LayerNorm is one of those architectural choices that looks tiny on paper, just dropping two operations. But it shapes the cost and stability profile of every modern transformer language model. Llama, Mistral, Gemma, Qwen, DeepSeek: every frontier open-weight family uses RMSNorm. The original 2017 transformer and BERT used LayerNorm. T5 already used an RMS-style norm in 2019, Llama 1 made RMSNorm the open-weight default in early 2023, and the shift has not reversed.

The swap is safe at pretraining time but not at inference time.

The two formulas, side by side

Both LayerNorm and RMSNorm act per token on a d_model-wide activation vector x.

LayerNorm

\text{LN}(x) = \gamma \cdot \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta

where:

mu = mean(x) over the d_model axis.
sigma^2 = mean((x - mu)^2) over the d_model axis.
gamma, beta are learnable vectors of shape (d_model,).
eps is a small constant for numerical stability (typically 1e-5).

LayerNorm does three distinct operations: center (subtract mean), scale (divide by std), affine (multiply gamma plus bias).

RMSNorm

\text{RMSNorm}(x) = \gamma \cdot \frac{x}{\sqrt{\frac{1}{d}\sum x_i^2 + \epsilon}}

where:

The denominator is the RMS of x: square each element, take the mean, square root.
gamma is a learnable vector of shape (d_model,).
eps is the same numerical stability constant.
NO beta.

RMSNorm does two distinct operations: scale by RMS, multiply by gamma.

The algebraic relationship

If mu = 0, then sigma = RMS (since variance = mean((x - mu)^2) = mean(x^2) when mu = 0). So in that special case, LayerNorm's normalization step collapses to RMSNorm's. RMSNorm can be thought of as 'LayerNorm under the implicit assumption that x has zero mean'.

Transformer activations are roughly mean-centered to begin with (the input embeddings are initialized that way; downstream projections preserve this approximately). So the assumption holds well enough in practice that the centering step provides little benefit.

Why the dropped operations are empirically safe

Cost savings at scale

Why retrofitting fails

The landscape of normalization techniques

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Property	LayerNorm	RMSNorm
Mean centering	Yes (x - mu)	No
Scale normalization	Divide by std	Divide by RMS
Learned scale gamma	Yes	Yes
Additive bias beta	Yes	No
Reductions per call	2 (mean, var)	1 (mean of squares)
Parameters per norm	2 * d_model	1 * d_model
Modern LLM adoption	BERT, GPT-2, GPT-3	Llama 1-4, Mistral, Gemma, Qwen, DeepSeek

Real products, models, and research that use this idea.

Llama 4 Maverick uses RMSNorm pre-norm throughout, gamma per d_model dimension, no bias.
Mistral Large 2 and Mistral Small use RMSNorm consistent with the Llama family convention.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does retrofitting RMSNorm into a LayerNorm-trained model break the model?

The downstream projections (W_Q, W_K, W_V, FFN W_in) and biases have learned to expect mean-centered input from LayerNorm. Without centering, the projections see a shifted distribution and the bias terms encode incorrect offsets. The resulting Q, K, V vectors fall outside the regime the attention softmax was trained for. Quality collapses immediately.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Claiming RMSNorm and LayerNorm are equivalent or just a rename. They have different gradients and remove two specific operations from LayerNorm.

60 second bullets to scan on the way to the call.

Write down both formulas: LayerNorm and RMSNorm
List the two operations RMSNorm drops

Primary sources. Browse if you want the original framing.