Layer Normalization
Also known as: LayerNorm, RMSNorm
Per-sample, per-position normalization. The glue that keeps transformer training stable.
A normalization technique that rescales activations within a single sample across the feature dimension. Stabilizes transformer training. Modern variants like RMSNorm drop the mean-subtraction step for ~10% speedup.
In practice
Critical for understanding training stability. Senior interviews probe pre-norm vs post-norm and why LLaMA uses RMSNorm.
Related topics
Questions that mention this term
Related terms
Attention Mechanism
How a model decides which input tokens to weight when computing each output token.
Transformer
The attention-only neural architecture behind GPT, Claude, Gemini, and almost every modern LLM.
Context Window
The max number of tokens a model can attend to at once.
Mixture of Experts (MoE)
Scale model capacity by routing each token to a small subset of expert MLPs instead of using all of them.
RoPE (Rotary Position Embedding)
Position info injected by rotating Q and K vectors, easy to extend to longer contexts.
Multi-Head Attention (MHA)
Run several attention heads in parallel with different projections, then concat. This captures multiple relationship types per layer.