Name what wraps the attention sub-layer in every transformer block.

Flashcard·Easy·4.0 · 0·~30s

Attempt it

Two things wrap attention in every transformer block: a LayerNorm (or RMSNorm) and a residual connection. The standard pre-norm pattern is x + Attention(LN(x)), where LN is applied to the input first, then attention runs, then the result is added back to the original x. The residual lets gradients flow cleanly through dozens of layers, without it, deep transformers do not train. Modern decoder-only LLMs like Llama and Mistral use pre-norm with RMSNorm; the original 2017 transformer used post-norm with LayerNorm.

Your answer

Every transformer block has a standard wrapper around the attention sub-layer. Two pieces: 1. **A normalization layer** (LayerNorm in the original 2017 transformer; RMSNorm in modern Llama, Mistral, Gemma). It standardizes the per-token activation vector to a known scale before attention sees it. 2. **A residual connection**, the skip path that adds the original input back to the attention output: `x + Attention(LN(x))`. The modern pattern is **pre-norm**: normalize first, then run attention, then add to the residual. The original 2017 transformer used **post-norm**: run attention first, add residual, then normalize. Pre-norm is more stable for very deep stacks and is the default in every modern LLM. The residual is the load-bearing piece. Without it, gradients have to flow through every attention sub-layer in series, and they degrade across the stack. Networks beyond a handful of layers stop training. With the residual, the gradient can take the skip path directly back to early layers, which is what makes 70- to 100-layer transformers trainable at all. The normalization keeps activations in a stable range so attention's softmax does not saturate, which is the other half of why the wrapper is universal.

Two wrappers, always: a normalization and a residual connection. The modern pre-norm pattern is `output = x + Attention(LN(x))`. The input x goes through normalization (LayerNorm or RMSNorm), then attention, then the attention output is added back to the original x via the skip connection. The same wrapper pattern repeats for the FFN sub-layer: `output = x + FFN(LN(x))`. The normalization (`LN`) keeps per-token activation vectors at a stable scale before attention reads them. Without it, dot-product magnitudes drift, softmax saturates, gradients vanish on non-peak keys. LayerNorm centers and scales; RMSNorm drops the centering and the bias, keeping only the divide-by-RMS plus a learnable scale. Llama and Mistral use RMSNorm because it is cheaper with no measurable quality loss. The residual connection is the architectural reason transformers train at depth. Without `x + ...`, the only path for the gradient is back through every attention block in series, and deep transformers do not train. With the residual, the gradient has a clean identity-path back to earlier layers (essentially a constant-1 derivative through the skip), so 70-, 80-, 96-layer stacks are stable. Pre-norm vs post-norm matters: the original 2017 transformer used post-norm (`LN(x + Attention(x))`), which works but requires careful learning-rate warmup. Pre-norm is the modern default precisely because it allows training without warmup and scales better.

Quick reference

TL;DR

Every transformer block wraps attention in two things: a normalization (LayerNorm or RMSNorm) on the input and a residual skip that adds the attention output back to the original input.

Memory aid

Easy to grasp

Picture a really long assembly line where each station does something to a product, but the product is also carried past every station on a conveyor belt. Each station gets to see the product, do its little adjustment, and then add that adjustment to whatever was on the belt. If the station messes up, the original product still arrives at the end. That conveyor belt is the residual stream. And before any station touches the product, an inspector standardizes its size so the station gets predictable input, that is the normalization. Both wrappers exist because without them deep stacks of stations would either break the product completely or fail to train at all.

Key concepts

Residual connection LayerNorm / RMSNorm Transformer block

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Name the two wrappers, distinguish pre-norm and post-norm, explain why the residual is load-bearing for depth, note LayerNorm vs RMSNorm, give modern LLM examples.

Real-world examples

Real products, models, and research that use this idea.

Llama 4 Maverick uses pre-norm RMSNorm wrapping attention and FFN sub-layers across all 80+ blocks.
Mistral Large 2 ships pre-norm RMSNorm with sliding-window attention inside the wrapper.
GPT-2, GPT-3, GPT-4 all use pre-norm with LayerNorm in the original implementations.
The original 2017 Vaswani transformer used post-norm LayerNorm, which required learning-rate warmup to train stably.
DeepSeek V4 uses pre-norm RMSNorm wrapping MLA-style attention sub-layers.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWalk through how the gradient flows through a residual connection and explain why this prevents vanishing gradients.

QWhy does pre-norm allow training without LR warmup while post-norm needs it?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Forgetting the residual or treating it as 'just nice to have'. Without the residual, gradients vanish across stacked layers and the network does not train.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.