Complete the pre-norm transformer block recipe
The two blanks are attention (top) and FFN / MLP (bottom). Every modern transformer block is exactly these two sublayers, each residual wrapped and pre-normed.
Think of a transformer block as a two step kitchen station. At the first station, every word looks around at all the other words and gathers context. That's attention. At the second station, each word goes into a private booth and thinks about what it just learned. That's the feed-forward network. Around each station, two helpers do invisible work. One rescales the input so it's not too loud or too quiet (the LayerNorm). The other quietly photocopies the original input and adds it back at the end so nothing gets lost (the residual). Stack 32 of these stations and you have GPT-style language understanding. The recipe is exactly two ingredients in two slots: Attention, then FFN.
Detailed answer & concept explanation~4 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: name the two blanks, write the formula, explain why each sublayer is necessary, mention the canonical 2026 variants (GQA, SwiGLU), and finish with the parameter split.
Real products, models, and research that use this idea.
- Llama 3.1 8B's `LlamaDecoderLayer` in Hugging Face transformers literally implements these two lines, with `self.self_attn` and `self.mlp`.
- Mistral 7B and Mistral Large 3 use identical block structure with sliding-window attention swapped into the first sublayer.
- DeepSeek V4 uses MLA in the first sublayer and a SwiGLU FFN in the second; same recipe, different attention variant.
- GPT-2's reference implementation (Karpathy's nanoGPT) shows the same two line block; only the norm placement differs (it's pre-norm now, was post-norm in original).
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does attention come first and FFN second instead of the reverse?
QWhat changes in the recipe for an encoder-decoder model?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Filling in 'Softmax' or 'LayerNorm' for one of the blanks. The two sublayers are attention (cross-position mixing) and FFN (per-position transformation); softmax and norm are pieces inside those sublayers.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.