Which components make up a standard transformer block?

A **transformer block** is the repeating unit that the model stacks 12, 32, or 96 times to build up depth. Every block has the same four ingredients. First, multi-head attention lets each token in the sequence look at every other token and pull in relevant context. Heads work in parallel, each one learning a different relationship. Second, a **feed-forward network** (an MLP, typically two linear layers with an activation between them) processes each token independently. This is where most of the parameters live, despite attention getting most of the diagrams. Third, residual connections add the block's input back to its output. Without these, deep networks fail to train because gradients (the learning signal) vanish through dozens of layers. Fourth, **layer normalization** rescales activations to keep them in a stable range. Modern transformers use pre-norm (norm before each sublayer) rather than the original post-norm because it trains more reliably at scale.

**A transformer block is two sublayers, each wrapped in a residual and a layer norm.** The two sublayers are multi-head attention and a position-wise feed-forward network. That is the entire recipe. Written out for a single block: $$x \leftarrow x + \text{Attn}(\text{LN}(x)), \quad x \leftarrow x + \text{FFN}(\text{LN}(x))$$ Attention mixes information across the sequence dimension. The feed-forward sublayer processes each position independently, giving the model capacity to compute features for each token. The residual lets gradients flow cleanly through depth. The layer norm keeps activations in a stable range so the optimizer behaves. Modern variants (Llama, Mistral, Qwen) tweak the details. Pre-norm replaces post-norm. RMSNorm replaces full LayerNorm. SwiGLU replaces ReLU or GELU in the FFN. GQA replaces MHA in the attention sublayer. The block structure stays identical: attention, then FFN, each wrapped in a residual and a norm.