Which components make up a standard transformer block?
A transformer block stacks multi-head attention and a feed-forward network, each wrapped in a residual connection and a layer norm. Four ingredients, same across every variant.
Imagine an assembly line where each station takes a sentence, processes it, and hands it on. The first station, attention, lets every word in the sentence look at every other word and gather context. The second station, the feed-forward network, gives each word a chance to think privately about what it just learned. Around each station, a forklift quietly carries the original input through the back so nothing gets lost (that is the residual connection), and a quality inspector rescales the output so it stays in a healthy range (that is the layer norm). Stack 32 of these stations and you have GPT. The four ingredients never change; only their order and the size of each station varies.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The transformer block is the single most stable design in deep learning. Vaswani and colleagues defined it in 2017, and every frontier LLM in 2026 still uses the same two-sublayer, residual-wrapped, norm-stabilized template. What changed is everything inside the boxes: the attention variant, the normalization, the activation. What did not change is the block.
This question is the kind of softball an interviewer uses to find out whether you understand the architecture as a recipe or as a slogan. Naming attention is the floor. Naming the FFN, the residual, and the norm, then explaining where each one is doing real work, is the senior-level answer.
The two sublayers and what they each do
A transformer block has exactly two sublayers and they do different jobs.
Multi-head attention mixes information across the sequence dimension. Each token computes a query and looks at every other token's key and value to gather context. Without attention, the model has no way for a token's representation to depend on any other token.
The feed-forward network processes each token independently. It is a two-layer MLP with a non-linearity between, typically 4 x d_model wide in the hidden dimension. This is where the model stores the bulk of its memorized knowledge and where most of the per-token compute happens.
The division of labor is deliberate. Attention is the cross-sequence operation; FFN is the per-position computation. Stacking them alternates the two operations, which is what gives transformers their expressive power.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Component | Original (2017) | Modern (2026) |
|---|---|---|
| Attention | Multi-head attention (MHA) | GQA, MQA, or MLA for KV efficiency |
| FFN activation | ReLU | SwiGLU or GeGLU |
| Normalization | LayerNorm (post-norm) | RMSNorm (pre-norm) |
| Residuals | Two per block | Two per block (unchanged) |
Real products, models, and research that use this idea.
- Llama 4 Maverick stacks transformer blocks with RMSNorm, SwiGLU FFN, and GQA attention; the block recipe matches the original 2017 template.
- OpenAI's GPT-5.5 and the o-series are decoder-only transformer stacks; exact hyperparameters are unpublished but the block structure is the standard one.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the FFN sublayer 'position-wise' and what does that buy you?
Each token is processed independently through the same MLP, which makes the operation trivially parallel and lets the attention sublayer carry all the cross-token mixing.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating attention as the entire transformer block. The feed-forward network carries about two-thirds of the parameters and most of the per-token compute.
60 second bullets to scan on the way to the call.
The four ingredients of a transformer block
Why residual connections are non-optional at depth
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.