**RMSNorm has a single learnable scale and a small numerical-stability constant.** The full formula is:
$$\text{RMSNorm}(x) = \gamma \cdot \frac{x}{\sqrt{\text{mean}(x^2) + \epsilon}}$$
The mean of squared elements is taken along the feature axis. Taking the square root gives the **root mean square** of x, hence the name. Dividing x by that quantity rescales it so the typical magnitude is 1.
Gamma is a learnable vector with one entry per model-width feature that gives the model a per-feature scale knob after normalization. Without it, every normalized output would have the same magnitude, which would limit expressivity.
Epsilon is a tiny constant, typically around 1e-5 or 1e-6, added inside the square root to prevent division by zero when x is exactly all zeros. It does not affect normal operation; it is purely a numerical safeguard.
Notice what is missing relative to LayerNorm. There is no mean subtraction (so RMSNorm does not recenter), and there is no learnable bias term. Both omissions are deliberate: they make RMSNorm cheaper and have been shown empirically to lose no quality.
**The two missing terms are `gamma` (the learnable per-feature scale) and `eps` (a small numerical-stability constant inside the square root).** The complete formula is:
$$\text{RMSNorm}(x) = \gamma \cdot \frac{x}{\sqrt{\text{mean}(x^2) + \epsilon}}$$
The gamma is a learnable vector of shape `(d_model,)` initialized to all ones; it gives the model one scale parameter per feature to recover the post-norm magnitude it wants. The eps is a fixed scalar, typically `1e-5` or `1e-6`, that prevents the denominator from being zero when `x` is exactly the zero vector. Llama uses `1e-5`; some implementations use `1e-6` or `1e-8`.
**Compared to LayerNorm**, RMSNorm drops two things. First, it drops the mean subtraction: LayerNorm subtracts `mean(x)` before normalizing, RMSNorm does not. Second, it drops the learnable bias `beta`: LayerNorm has both a scale `gamma` and a bias `beta`, RMSNorm has only the scale.
**Why those omissions are safe.** Zhang and Sennrich (2019) showed empirically that for transformer blocks, the mean-subtraction step contributes essentially nothing to final quality; what mattered was the rescaling. Dropping the mean subtraction saves one reduction pass over the feature axis. Dropping the bias saves a small number of parameters per layer. Together, RMSNorm is roughly 30 to 50 percent faster than LayerNorm on modern GPUs and has been adopted in Llama 1, 2, 3, Mistral, Qwen, and DeepSeek decoder LLMs.
Quick reference
TL;DR
RMSNorm(x) = gamma * x / sqrt(mean(x^2) + eps). One learnable scale `gamma` per feature, and a small `eps` (typically 1e-5 or 1e-6) under the square root for numerical stability.
Imagine each token in the model has a list of 4096 numbers. Some lists are bigger overall, some smaller. RMSNorm just measures the typical size of those numbers (the root mean square) and divides everything by that, so every list ends up at roughly the same overall scale. Then it lets the model learn one knob per position to fine-tune the final scale. That is the whole operation. The two missing pieces of the puzzle are the knob name (gamma) and the tiny safety constant under the square root (epsilon) that stops the division from blowing up when the input is all zeros.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Architectures where mean subtraction is empirically needed (some vision transformer variants, batch-normalized CNNs) RMSNorm assumes the mean of the activation along the feature axis is uninformative; that assumption holds well for transformer residual streams but is not universal.
Reproducing an exact LayerNorm baseline for comparison If your goal is matching prior LayerNorm-based results exactly (e.g., reproducing GPT-2 or BERT), you must use LayerNorm; RMSNorm is not bit-identical.
Targets without an optimized RMSNorm kernel If your runtime has a fused LayerNorm kernel but no fused RMSNorm path (some older accelerators, mobile NPUs), the per-step compute win can flip and LayerNorm becomes faster on wall-clock.
Quick scan · 2–4 min
Everything important, quickly.
2 min: write the formula, identify gamma and eps as the two blanks, contrast with LayerNorm (no mean subtraction, no bias), note the compute saving, and name eps inside the radical.
Side-by-side
Aspect
LayerNorm
RMSNorm
Normalizing quantity
`(x - mean(x)) / sqrt(var(x) + eps)`
`x / sqrt(mean(x^2) + eps)`
Mean subtraction
Yes
No
Learnable params per feature
2 (gamma, beta)
1 (gamma)
Reductions over feature axis
2 (mean, then variance)
1 (mean of x^2)
Typical eps
`1e-5`
`1e-5` or `1e-6`
Compute cost
Baseline
Roughly 30-50% faster
Real-world examples
Real products, models, and research that use this idea.
1Llama 3.1 8B and 70B both use RMSNorm with eps `1e-5` in every block (two per block, plus a final RMSNorm before the unembedding head).
2Mistral Large and Mixtral 8x22B use RMSNorm with eps `1e-5`.
3PyTorch added `torch.nn.RMSNorm` as a first-class module in PyTorch 2.4 (2024), matching the Llama-style inside-eps formula.
4T5 (Raffel et al., 2020) was one of the first major models to use RMSNorm in production, predating its dominance in decoder LLMs.
5Llama's reference implementation in the `llama` repo defines RMSNorm as `x * rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * weight`, which exactly matches the formula.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is gamma per-feature (shape `(d_model,)`) rather than a single scalar?
Sign in or go Pro to see the approach for this follow-up.
QWhat happens if you set eps too large, like `1e-2`?
Sign in or go Pro to see the approach for this follow-up.
QHow does the inside-eps versus outside-eps placement matter for gradient flow?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Adding a learnable bias term. RMSNorm deliberately omits the bias that LayerNorm has; only one learnable parameter per feature.