Complete the pre-norm transformer block recipe

Attempt it

x = x + (LN(x)) # attention sublayer x = x + (LN(x)) # feed-forward sublayer

Quick reference

TL;DR

The two blanks are attention (top) and FFN / MLP (bottom). Every modern transformer block is exactly these two sublayers, each residual wrapped and pre-normed.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Think of a transformer block as a two step kitchen station. At the first station, every word looks around at all the other words and gathers context. That's attention. At the second station, each word goes into a private booth and thinks about what it just learned. That's the feed-forward network. Around each station, two helpers do invisible work. One rescales the input so it's not too loud or too quiet (the LayerNorm). The other quietly photocopies the original input and adds it back at the end so nothing gets lost (the residual). Stack 32 of these stations and you have GPT-style language understanding. The recipe is exactly two ingredients in two slots: Attention, then FFN.

Key concepts

Transformer architecture Attention

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: name the two blanks, write the formula, explain why each sublayer is necessary, mention the canonical 2026 variants (GQA, SwiGLU), and finish with the parameter split.

Real-world examples

Real products, models, and research that use this idea.

Llama 3.1 8B's `LlamaDecoderLayer` in Hugging Face transformers literally implements these two lines, with `self.self_attn` and `self.mlp`.
Mistral 7B and Mistral Large 3 use identical block structure with sliding-window attention swapped into the first sublayer.
DeepSeek V4 uses MLA in the first sublayer and a SwiGLU FFN in the second; same recipe, different attention variant.
GPT-2's reference implementation (Karpathy's nanoGPT) shows the same two line block; only the norm placement differs (it's pre-norm now, was post-norm in original).

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does attention come first and FFN second instead of the reverse?

Sign in or go Pro to see the approach for this follow-up.

QWhat changes in the recipe for an encoder-decoder model?

Sign in or go Pro to see the approach for this follow-up.

QCould you write a single sublayer transformer where attention and FFN are merged?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Filling in 'Softmax' or 'LayerNorm' for one of the blanks. The two sublayers are attention (cross-position mixing) and FFN (per-position transformation); softmax and norm are pieces inside those sublayers.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.