Complete the pre-norm transformer block recipe
The two blanks are attention (top) and FFN / MLP (bottom). Every modern transformer block is exactly these two sublayers, each residual wrapped and pre-normed.
Think of a transformer block as a two step kitchen station. At the first station, every word looks around at all the other words and gathers context. That's attention. At the second station, each word goes into a private booth and thinks about what it just learned. That's the feed-forward network. Around each station, two helpers do invisible work. One rescales the input so it's not too loud or too quiet (the LayerNorm). The other quietly photocopies the original input and adds it back at the end so nothing gets lost (the residual). Stack 32 of these stations and you have GPT-style language understanding. The recipe is exactly two ingredients in two slots: Attention, then FFN.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The pre-norm transformer block is a two line formula. Almost every other architectural decision in a modern LLM happens inside those two lines: which attention variant goes in the first slot, which activation goes in the FFN. The recipe itself is so stable that filling in these two blanks correctly is the first thing a Llama codebase asks you to do.
This question tests whether you have internalized the block as a recipe with two sublayers. Anything else (softmax, normalization, residual) is either part of one of those sublayers or part of the wrap around them.
The two sublayers and why they exist
A transformer block has exactly two sublayers: attention and feed-forward network. They do different jobs and the model needs both.
Attention is the cross-sequence operation. Each token computes a query vector, looks at every other token's key and value vector, and gathers a weighted sum of those values. This is the only operation in the block that mixes information across positions. Take attention out and a transformer reduces to a per-token MLP applied identically at every position, with no context at all; useless for language modeling.
FFN is the per-position computation. Each token is processed independently through a two or three layer MLP. The hidden dimension is typically 3-4x d_model. The FFN gives the block capacity to compute features that don't depend on other tokens (things like 'is this token a verb?' or 'is this token capitalized?'). Take the FFN out and the model loses most of its memorization capacity; pure attention stacks lose 1-3 points of perplexity at matched parameter count.
The interplay is what makes transformers work. Attention mixes; FFN computes. Alternating them across 32 or 80 layers produces the deep, contextualized features that drive language modeling.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3.1 8B's `LlamaDecoderLayer` in Hugging Face transformers literally implements these two lines, with `self.self_attn` and `self.mlp`.
- Mistral 7B and Mistral Large 3 use identical block structure with sliding-window attention swapped into the first sublayer.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does attention come first and FFN second instead of the reverse?
Attention first gives the FFN contextualized features to work on. FFN first would make per-token features uninformed by sequence context. Every published ablation since 2017 confirms this order.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Filling in 'Softmax' or 'LayerNorm' for one of the blanks. The two sublayers are attention (cross-position mixing) and FFN (per-position transformation); softmax and norm are pieces inside those sublayers.
60 second bullets to scan on the way to the call.
The two sublayers that make up a transformer block
Why attention comes first and FFN second
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.