Layer Normalization
Also known as: LayerNorm, RMSNorm
Per-sample, per-position normalization. The glue that keeps transformer training stable.
A normalization technique that rescales activations within a single sample across the feature dimension. Stabilizes transformer training. Modern variants like RMSNorm drop the mean-subtraction step for ~10% speedup.
In practice
Critical for understanding training stability. Senior interviews probe pre-norm vs post-norm and why LLaMA uses RMSNorm.
Related topics
Questions that mention this term
- RMSNorm versus LayerNorm, what is kept and what is dropped?
- Walk through why pre-norm trains stably past 100 layers while post-norm needs warmup tricks
- Name what wraps the attention sub-layer in every transformer block.
- Pre-norm versus post-norm: which placement makes deep stacks stable?
- Why does normalizing Q and K before the dot product help training at scale?
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Causal Mask
Attention mask that hides future tokens so each position can only see itself and prior tokens.
Context Window
The max number of tokens a model can attend to at once.
Decoder-Only
Single autoregressive transformer stack: the shape of every modern frontier LLM.
Encoder-Decoder
Transformer with separate encoder + decoder stacks; strong for translation and structured seq2seq tasks.
Mixture of Experts (MoE)
Scale model capacity by routing each token to a small subset of expert MLPs instead of using all of them.