Which components make up a standard transformer block? · Zenaique

Attempt it

Quick reference

TL;DR

A transformer block stacks multi-head attention and a feed-forward network, each wrapped in a residual connection and a layer norm. Four ingredients, same across every variant.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine an assembly line where each station takes a sentence, processes it, and hands it on. The first station, attention, lets every word in the sentence look at every other word and gather context. The second station, the feed-forward network, gives each word a chance to think privately about what it just learned. Around each station, a forklift quietly carries the original input through the back so nothing gets lost (that is the residual connection), and a quality inspector rescales the output so it stays in a healthy range (that is the layer norm). Stack 32 of these stations and you have GPT. The four ingredients never change; only their order and the size of each station varies.

Key concepts

Transformer architecture Attention

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: four ingredients + pre-norm vs post-norm + FFN dominates parameter count + RMSNorm and SwiGLU as 2026 swaps + how GQA/MLA modify attention without changing the block recipe.

Side-by-side

Component	Original (2017)	Modern (2026)
Attention	Multi-head attention (MHA)	GQA, MQA, or MLA for KV efficiency
FFN activation	ReLU	SwiGLU or GeGLU
Normalization	LayerNorm (post-norm)	RMSNorm (pre-norm)
Residuals	Two per block	Two per block (unchanged)

Real-world examples

Real products, models, and research that use this idea.

Llama 4 Maverick stacks transformer blocks with RMSNorm, SwiGLU FFN, and GQA attention; the block recipe matches the original 2017 template.
OpenAI's GPT-5.5 and the o-series are decoder-only transformer stacks; exact hyperparameters are unpublished but the block structure is the standard one.
Mistral Large 3 ships pre-norm, SwiGLU FFN, and GQA in every block, with sliding-window attention as a per-layer modification.
DeepSeek V4 uses MLA (Multi-head Latent Attention) inside an otherwise standard block, demonstrating that the attention sublayer can be swapped without changing the recipe.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the FFN sublayer 'position-wise' and what does that buy you?

Sign in or go Pro to see the approach for this follow-up.

QWhat changes if you remove the layer norm before each sublayer?

Sign in or go Pro to see the approach for this follow-up.

QHow does swapping MHA for GQA inside the block affect downstream behavior?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating attention as the entire transformer block. The feed-forward network carries about two-thirds of the parameters and most of the per-token compute.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.