Complete the RMSNorm formula

Fill in blank·Easy·4.0 · 0·~1 min·Relevant atMeta Mistral AI

Attempt it

RMSNorm(x) = * x / sqrt(mean(x^2) + )

**RMSNorm has a single learnable scale and a small numerical-stability constant.** The full formula is: $$\text{RMSNorm}(x) = \gamma \cdot \frac{x}{\sqrt{\text{mean}(x^2) + \epsilon}}$$ The mean of squared elements is taken along the feature axis. Taking the square root gives the **root mean square** of x, hence the name. Dividing x by that quantity rescales it so the typical magnitude is 1. Gamma is a learnable vector with one entry per model-width feature that gives the model a per-feature scale knob after normalization. Without it, every normalized output would have the same magnitude, which would limit expressivity. Epsilon is a tiny constant, typically around 1e-5 or 1e-6, added inside the square root to prevent division by zero when x is exactly all zeros. It does not affect normal operation; it is purely a numerical safeguard. Notice what is missing relative to LayerNorm. There is no mean subtraction (so RMSNorm does not recenter), and there is no learnable bias term. Both omissions are deliberate: they make RMSNorm cheaper and have been shown empirically to lose no quality.

**The two missing terms are `gamma` (the learnable per-feature scale) and `eps` (a small numerical-stability constant inside the square root).** The complete formula is: $$\text{RMSNorm}(x) = \gamma \cdot \frac{x}{\sqrt{\text{mean}(x^2) + \epsilon}}$$ The gamma is a learnable vector of shape `(d_model,)` initialized to all ones; it gives the model one scale parameter per feature to recover the post-norm magnitude it wants. The eps is a fixed scalar, typically `1e-5` or `1e-6`, that prevents the denominator from being zero when `x` is exactly the zero vector. Llama uses `1e-5`; some implementations use `1e-6` or `1e-8`. **Compared to LayerNorm**, RMSNorm drops two things. First, it drops the mean subtraction: LayerNorm subtracts `mean(x)` before normalizing, RMSNorm does not. Second, it drops the learnable bias `beta`: LayerNorm has both a scale `gamma` and a bias `beta`, RMSNorm has only the scale. **Why those omissions are safe.** Zhang and Sennrich (2019) showed empirically that for transformer blocks, the mean-subtraction step contributes essentially nothing to final quality; what mattered was the rescaling. Dropping the mean subtraction saves one reduction pass over the feature axis. Dropping the bias saves a small number of parameters per layer. Together, RMSNorm is roughly 30 to 50 percent faster than LayerNorm on modern GPUs and has been adopted in Llama 1, 2, 3, Mistral, Qwen, and DeepSeek decoder LLMs.

Quick reference

TL;DR

RMSNorm(x) = gamma * x / sqrt(mean(x^2) + eps). One learnable scale `gamma` per feature, and a small `eps` (typically 1e-5 or 1e-6) under the square root for numerical stability.

Memory aid

Easy to grasp

Imagine each token in the model has a list of 4096 numbers. Some lists are bigger overall, some smaller. RMSNorm just measures the typical size of those numbers (the root mean square) and divides everything by that, so every list ends up at roughly the same overall scale. Then it lets the model learn one knob per position to fine-tune the final scale. That is the whole operation. The two missing pieces of the puzzle are the knob name (gamma) and the tiny safety constant under the square root (epsilon) that stops the division from blowing up when the input is all zeros.

Key concepts

Transformer architecture

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

2 min: write the formula, identify gamma and eps as the two blanks, contrast with LayerNorm (no mean subtraction, no bias), note the compute saving, and name eps inside the radical.

Side-by-side

Aspect	LayerNorm	RMSNorm
Normalizing quantity	`(x - mean(x)) / sqrt(var(x) + eps)`	`x / sqrt(mean(x^2) + eps)`
Mean subtraction	Yes	No
Learnable params per feature	2 (gamma, beta)	1 (gamma)
Reductions over feature axis	2 (mean, then variance)	1 (mean of x^2)
Typical eps	`1e-5`	`1e-5` or `1e-6`
Compute cost	Baseline	Roughly 30-50% faster

Real-world examples

Real products, models, and research that use this idea.

Llama 3.1 8B and 70B both use RMSNorm with eps `1e-5` in every block (two per block, plus a final RMSNorm before the unembedding head).
Mistral Large and Mixtral 8x22B use RMSNorm with eps `1e-5`.
PyTorch added `torch.nn.RMSNorm` as a first-class module in PyTorch 2.4 (2024), matching the Llama-style inside-eps formula.
T5 (Raffel et al., 2020) was one of the first major models to use RMSNorm in production, predating its dominance in decoder LLMs.
Llama's reference implementation in the `llama` repo defines RMSNorm as `x * rsqrt(x.pow(2).mean(-1, keepdim=True) + eps) * weight`, which exactly matches the formula.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is gamma per-feature (shape `(d_model,)`) rather than a single scalar?

QWhat happens if you set eps too large, like `1e-2`?

QHow does the inside-eps versus outside-eps placement matter for gradient flow?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Adding a learnable bias term. RMSNorm deliberately omits the bias that LayerNorm has; only one learnable parameter per feature.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.