Full multi-head attention with one KV pair per query head
Sinusoidal absolute positional encoding added at input
Tied input embedding and output unembedding
The original 2017 transformer paper and a 2026 Llama 4 block use the same overall shape: two sublayers (attention and feed-forward), each wrapped in a residual connection and a normalization layer. What changed is the contents of each box.
**Normalization** swapped from LayerNorm to RMSNorm, and the placement moved from post-norm (after the residual add) to pre-norm (before the sublayer). These are two independent changes that happened around the same time.
**The FFN activation** went from ReLU to SwiGLU, a gated unit that uses three matrices instead of two. SwiGLU consistently improves quality enough that every modern Llama, Mistral, and Qwen model ships it.
**Attention** changed from full multi-head (one KV pair per query head) to GQA, where multiple query heads share fewer KV heads. This shrinks the KV cache at decode time, which is the biggest serving cost lever at long context.
**Positional encoding** moved from sinusoidal addition at the input to RoPE rotation inside every attention layer.
**Embeddings** were tied in 2017 to save parameters; at frontier scale 2026 models often untie them for a small quality gain.
**The recipe is constant, the ingredients changed.** Vaswani 2017 and a 2026 Llama 4 dense block share the same two sublayer, residual wrapped, norm stabilized template, but every component inside that template has been swapped.
The six canonical swaps:
- **LayerNorm to RMSNorm**: drop the mean subtraction and bias term; keep only RMS rescaling. Cheaper per token, no measurable quality loss.
- **Post-norm to pre-norm**: apply the norm BEFORE each sublayer rather than after the residual add. Pre-norm trains stably past 100 layers; post-norm needs careful warmup tricks past 24.
- **ReLU to SwiGLU**: the FFN activation becomes a gated linear unit with three weight matrices. To preserve total parameters, hidden dim shrinks from `4 * d_model` to roughly `2.67 * d_model`.
- **MHA to GQA**: multiple query heads share fewer K and V heads. Llama 3.1 8B uses 32 query heads with 8 KV heads, shrinking the KV cache 4x.
- **Sinusoidal PE to RoPE**: instead of adding a position vector at the input, rotate Q and K by position-dependent angles inside each attention layer. The dot product `q_m · k_n` then depends only on `m - n`.
- **Tied embeddings to untied**: at frontier scale, the extra `vocab * d_model` parameters are worth the small quality bump.
None of these change the block recipe; together they define every modern open weight LLM.
Quick reference
TL;DR
Six swaps separate 2017 Vaswani from a 2026 Llama 4 block: RMSNorm, pre-norm, SwiGLU, GQA, RoPE, and untied embeddings. The recipe is unchanged, but every ingredient has moved.
Picture a 2017 transformer block as a kitchen with six labeled jars. Nine years later the kitchen layout is identical (same counters, same two cooking stations, same two storage shelves) but every jar has been replaced. Salt got swapped for a fancier seasoning that's cheaper to use. The bread got upgraded from sourdough to brioche. The old measuring cups got traded for a clever system that takes up less space without losing accuracy. The blueprint is the same recipe on the wall; the pantry is unrecognizable. That's exactly what happened to the transformer between Vaswani's original paper and Llama 4. You still have two sublayers wrapped in residuals and norms, but normalization, activation, attention, positional encoding, and word table sharing all moved to better defaults.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Teaching the transformer recipe to a complete beginner Walking through six swaps at once obscures the unchanged block structure. Establish the two-sublayer recipe first, then layer in the swaps.
Comparing a 2017-era architecture to a niche 2026 variant like Mamba or Hyena These architectures abandon the transformer block entirely; mapping their components onto the six swap table is a category error.
Auditing a small (sub-1B) educational model Several swaps (untied embeddings, GQA) have weaker justification at small scale; the 2017 defaults may still be appropriate.
Quick scan · 2–4 min
Everything important, quickly.
7 min: walk the six swaps, separate the normalization axis from the attention axis, explain each swap's motivation, name the 2026 frontier models that adopted them, and finish with the unchanged block recipe.
Side-by-side
Axis
2017 Vaswani
2026 Llama 4 / Mistral Large 3
Normalization
LayerNorm + post-norm
RMSNorm + pre-norm
FFN activation
ReLU, hidden = 4 x d_model
SwiGLU, hidden ~ 2.67 x d_model
Attention
MHA, one KV pair per query head
GQA, 4-8 query heads per KV head
Positional encoding
Sinusoidal, added at input
RoPE, rotates Q/K in each layer
Embeddings
Tied input and output
Often untied at frontier scale
Real-world examples
Real products, models, and research that use this idea.
1Llama 4 Maverick (MoE 400B/17B-active) ships all six modern defaults: RMSNorm, pre-norm, SwiGLU, GQA, RoPE, untied embeddings.
2Mistral Large 3 uses the same six defaults plus sliding-window attention as a per-layer modification.
4Qwen 3.5 and Gemma 4 both inherit the same six swap baseline; their differentiation is in MoE routing, context length, and tokenizer choices.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy did RMSNorm not appear until years after pre-norm became standard?
Sign in or go Pro to see the approach for this follow-up.
QCould you mix swaps freely (say, SwiGLU with sinusoidal PE and full MHA)?
Sign in or go Pro to see the approach for this follow-up.
QWhat would a seventh swap look like in 2027?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Pairing post-norm with RMSNorm or pairing RoPE with sinusoidal PE. Pre-norm vs post-norm is independent of LayerNorm vs RMSNorm; sinusoidal lives at the input, RoPE lives inside attention.