**Option B is right.** The reason transformers use LayerNorm or RMSNorm is structural, not about cost or residuals or positions.
### What each normalization actually normalizes over
**BatchNorm** computes a mean and variance for each channel ACROSS the batch dimension. For a CNN with input shape `(batch, channels, H, W)`, it averages over batch and spatial positions to get one mean and variance per channel.
**LayerNorm** computes mean and variance for each example ACROSS the feature dimension. For a transformer with input shape `(batch, seq_len, d_model)`, it averages over d_model to get one mean and variance per token position per example. Each token is normalized using only its own features.
**RMSNorm** is the same as LayerNorm but drops the mean subtraction and the learned bias. Divide by sqrt of mean-square along d_model, scale by a learned per-channel weight.
### Why BatchNorm fails for transformers
1. **Variable-length sequences need padding.** Real text sequences have different lengths, so batches are padded to the longest in the batch. Those padded positions are noise. If BatchNorm averages over padded positions, the padding pollutes the mean and variance for every real token. Masking them out helps but is fiddly and slow.
2. **Autoregressive inference uses batch size 1.** When generating one new token at a time for a single user, batch size is 1. BatchNorm with batch=1 has no batch statistic to compute; it has to fall back to running averages from training, which generalize poorly.
3. **Batch composition leaks information across examples.** Token i in example A is normalized using statistics that include tokens from examples B, C, and D. That is correlation across the batch, which is exactly what you do not want when each user's generation should be independent.
### Why the other options are wrong
**Option A (cost)** is wrong. The kernel cost difference between BatchNorm and LayerNorm is small and not what drives the choice.
**Option C (residuals)** is wrong. ResNets use BatchNorm with residual connections every day. BatchNorm and residuals are perfectly compatible.
**Option D (positional information)** is wrong. Neither LayerNorm nor BatchNorm carries positional information. Position comes from the positional encoding (sinusoidal, learned absolute, or RoPE), never from normalization.
**Option B is the answer.** BatchNorm's statistics are computed across the batch axis. LayerNorm and RMSNorm compute their statistics within a single token along `d_model`. The axis of normalization is the load-bearing distinction.
### Three concrete failure modes of BatchNorm in a transformer
**1. Padding contaminates batch statistics.** Variable-length text batches are padded to a common max length. BatchNorm averages over batch and sequence positions, so padded positions shift the mean and inflate the variance for every real token. Manual masking helps but is fiddly and slow. LayerNorm sidesteps this because it normalizes per-token along d_model, never crossing the batch or sequence dimension.
**2. Autoregressive inference at batch size 1 breaks BatchNorm.** Serving one user's chat completion runs at batch=1. BatchNorm has no meaningful batch statistic there, so it falls back to training-time running averages that reflect a different distribution. LayerNorm is invariant to batch size; the same statistic is computed at batch=1 or batch=8192.
**3. Batch composition leaks across examples.** In BatchNorm, token i in example A is normalized using statistics that include tokens from examples B, C, D. Two inputs that were independent become correlated through normalization, unwanted for generative serving where each user's output should be statistically independent.
### Why each wrong option is wrong
**A (cost)**: small and not the deciding factor; RMSNorm shows even cheaper LayerNorm variants exist. **C (residuals)**: false; ResNet uses BatchNorm with residual connections in every block. **D (positional info)**: false; position lives in the positional encoding (sinusoidal, learned absolute, ALiBi, RoPE), never in normalization.
The principle: normalize along an axis you can trust. CNNs on fixed-size images can trust the batch axis; transformers on variable-length text with batch=1 inference cannot.
Quick reference
TL;DR
BatchNorm's statistics depend on the batch axis, which is broken by padding and by batch-size-1 inference. LayerNorm normalizes per token along d_model and is immune to both.
Picture a classroom where you grade each student by comparing them to the rest of the class that day. If the class is empty, you cannot grade anyone. If half the class is absent and the seats are filled with cardboard cutouts to keep the room looking full, the cutouts drag the class average down and now every real student looks above average. That is BatchNorm in a transformer: the batch is the 'class', padded positions are the cutouts, and a batch of one at inference time is the empty room. LayerNorm grades each student by comparing them to their own internal features (height, weight, age, scores), never looking at the rest of the class. It works with one student, with full classes, with cardboard cutouts, with anything. That self-contained grading is exactly what an autoregressive language model needs because it generates one new token at a time and the batch composition changes constantly.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Training a vision CNN on fixed-size images with large stable batches BatchNorm is the right default there. The batch axis is reliable, the implicit batch-noise regularization helps, and inference batches look like training batches.
Training with extremely small batches (under 8) on any architecture BatchNorm degrades sharply at small batch size. GroupNorm or LayerNorm is the standard replacement.
Building a normalization-free transformer (recent research direction) Approaches like DyT (Dynamic Tanh) replace LayerNorm with a parameterized activation. Still a research direction in 2026; not production-default.
Quick scan · 2–4 min
Everything important, quickly.
3 min: contrast the axis BatchNorm normalizes over with the axis LayerNorm normalizes over, give the three concrete failure modes (variable-length padding, batch=1 inference, cross-example leakage), and refute each wrong option with one sentence.
Side-by-side
Property
BatchNorm
LayerNorm
RMSNorm
Axis normalized over
Batch + spatial/sequence
d_model (per token)
d_model (per token)
Subtracts mean?
Yes
Yes
No
Has learned bias (beta)?
Yes
Yes
No
Robust to variable-length sequences?
No (padding contaminates)
Yes
Yes
Robust to batch size 1?
No (uses running stats)
Yes
Yes
Train-time vs inference-time stats?
Different (EMA running stats)
Identical
Identical
Typical use
Vision CNNs
Transformers, RNNs
Modern LLM transformers
Real-world examples
Real products, models, and research that use this idea.
1Llama, Mistral, Qwen, DeepSeek all use RMSNorm (the cheaper LayerNorm cousin) and would refuse to ship BatchNorm in their attention blocks for exactly the variable-length and batch-1 inference reasons above.
2BERT and the original 2017 transformer use LayerNorm. The decision to avoid BatchNorm predates the modern decoder-only era by years.
3ResNet50 (the canonical CNN backbone) uses BatchNorm with residual connections in every bottleneck block. This directly refutes the 'BatchNorm cannot represent residuals' option.
4vLLM and TGI inference stacks routinely serve a single user's request at batch=1 (especially in chat completion mode). LayerNorm and RMSNorm are invariant to this; BatchNorm would be running on stale training-time statistics.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy did the field move from LayerNorm to RMSNorm in modern LLMs?
Sign in or go Pro to see the approach for this follow-up.
QIf you had to use BatchNorm in a transformer, how would you make it work?
Sign in or go Pro to see the approach for this follow-up.
QPre-norm or post-norm: does the placement choice change which normalization to use?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Calling BatchNorm 'too expensive' for transformers. Cost is not the issue. The issue is that BatchNorm's statistics are contaminated by padding and break entirely at batch size 1 during autoregressive inference.