The 2017 Vaswani transformer is a recognizable ancestor of modern LLMs but four of its ingredients were swapped between 2018 and 2024: post-norm became pre-norm, sinusoidal PE became RoPE, ReLU became SwiGLU, and
Think of the 2017 transformer like a 2017 sports car. Same number of wheels, same shape of the body, same idea of how the engine sits up front. But over the next ten years they swapped the fuel injection (sinusoidal positions became RoPE), the engine block (ReLU became SwiGLU), where the cooling system sits (post-norm became pre-norm), and how many cylinders share a spark plug (MHA became GQA). The car still looks like a car. The two-seat layout, the chassis, the four-wheel arrangement, those did not change. That is the block recipe. The interview trap option says the chassis got rebuilt; it did not.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The 'what changed from 2017 to 2026' question is a recurring interview format because it tests two things at once: whether you have the modern decoder-LLM template memorized, and whether you can distinguish architectural-recipe stability from ingredient evolution. Most candidates can name two or three swaps. Senior candidates can name all four block-internal swaps PLUS articulate that the block recipe itself is unchanged.
The four real swaps
Post-norm became pre-norm. Original Vaswani placed LayerNorm AFTER each sublayer's residual: LayerNorm(x + Sublayer(x)). This is fine for a 6-layer model but becomes catastrophic at depth. The residual path is repeatedly renormalized, which compresses the activation magnitude and disrupts gradient flow. Pre-norm x + Sublayer(LayerNorm(x)) leaves the residual path as a clean identity sum and only normalizes the sublayer's input. Xiong et al. (2020) demonstrated this empirically and theoretically. Every 2026 decoder LLM is pre-norm.
Sinusoidal PE became RoPE. The 2017 paper added fixed sinusoidal vectors to the input embedding to inject position. RoPE (Su et al. 2021) rotates Q and K vectors inside the attention sublayer by a position-dependent angle. The result: the attention score Q · K^T depends on the RELATIVE position between query and key, not on an absolute position embedding added at the input. RoPE generalizes to longer contexts much better (with appropriate base-frequency tuning) and is the canonical choice for Llama, Mistral, Qwen, and DeepSeek.
ReLU became SwiGLU. Vaswani's FFN was Linear → ReLU → Linear. SwiGLU (Shazeer 2020) is Linear((W_up · x) * Swish(W_gate · x)), where Swish is the smooth sigmoid-times-x activation. Three matrices instead of two; gated multiplicative interaction; consistently better quality. Llama 1+, Mistral, Qwen, DeepSeek all ship SwiGLU.
MHA became GQA. Original MHA gave each head its own K and V. This blows up the KV cache at long context. GQA (Ainslie et al. 2023) groups multiple query heads to share one K/V head; shrinks the cache 4x to 8x. Llama 2 introduced it for the 70B; Llama 3 standardized it across sizes; Mistral and Qwen followed. DeepSeek V3 uses MLA as a related compression strategy.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Component | Vaswani 2017 | Modern 2026 (Llama/Mistral) |
|---|---|---|
| Block recipe | Attention + FFN, each residual-wrapped | Attention + FFN, each residual-wrapped (unchanged) |
| Norm placement | Post-norm | Pre-norm |
| Norm function | LayerNorm | RMSNorm |
| Positional encoding | Sinusoidal absolute, added to input | RoPE, applied inside attention |
| FFN activation | ReLU | SwiGLU |
| Attention variant | Multi-head attention | Grouped-query attention or MLA |
| Overall structure | Encoder-decoder | Decoder-only |
Real products, models, and research that use this idea.
- Llama 3.1 8B: pre-norm, RoPE (with NTK-aware scaling for 128k context), SwiGLU FFN, GQA (8 KV heads for 32 query heads), RMSNorm. Every Vaswani 2017 ingredient swapped.
- Mistral Large 3: pre-norm, RoPE, SwiGLU, GQA, RMSNorm. Same swap inventory.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf pre-norm is so much better than post-norm at depth, why did Vaswani 2017 use post-norm?
Their model was only 6 encoder + 6 decoder layers. At that depth, post-norm is fine; the instability shows up at 24+ layers. The 2017 design was not wrong for the 2017 scale; it became wrong once depth scaled up.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Including the block recipe itself as 'no longer standard'. It is the single most stable design in deep learning; every modern LLM still uses the attention then FFN two-sublayer template wrapped in residuals.
60 second bullets to scan on the way to the call.
Post-norm versus pre-norm placement and why pre-norm wins at depth
Contrast sinusoidal PE with RoPE on injection location, mechanism, and long-context generalization
Primary sources. Browse if you want the original framing.
- Vaswani et al., Attention Is All You Need (2017)
- Xiong et al., On Layer Normalization in the Transformer Architecture (2020)
- Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding (2021)
- Shazeer, GLU Variants Improve Transformer (2020)
- Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023)
Same topic, related formats. Practice these next.