Which design choices from the original 'Attention Is All You Need' are no longer standard?

Multi-select·Medium·4.0 · 0·~1 min·Asked atMistral AI·Relevant atAnthropic Google Meta

Attempt it

**The four that changed.** **Post-norm became pre-norm.** Vaswani 2017 placed the LayerNorm AFTER each sublayer's residual: `LayerNorm(x + Sublayer(x))`. Modern LLMs place it BEFORE the sublayer: `x + Sublayer(LayerNorm(x))`. Pre-norm trains much more stably at depth. **Sinusoidal absolute PE became RoPE.** The original paper used fixed sinusoidal vectors added to the input embedding to inject position information. Llama, Mistral, Qwen, DeepSeek, and most other 2026 decoder LLMs use **Rotary Position Embedding** (Su et al., 2021), which rotates Q and K vectors by a position-dependent angle inside the attention sublayer. **ReLU became SwiGLU.** The original FFN was `Linear → ReLU → Linear`. Modern LLMs use SwiGLU: `(Linear(x) * Swish(Linear(x))) → Linear`. Three matrices instead of two, but consistently better quality per parameter. **Full MHA became GQA.** Vaswani used multi-head attention with independent K and V per head. GQA (Grouped Query Attention; Ainslie et al., 2023) lets multiple query heads share one K/V head, shrinking the KV cache 4x to 8x at decode time without quality loss. **The one that did NOT change.** The two-sublayer block recipe (attention then FFN, each residual-wrapped) is unchanged. Every modern decoder LLM has exactly this structure. The ingredients inside got swapped; the recipe stayed.

**The four 2017 choices that are no longer standard.** 1. **Post-norm placement.** Vaswani 2017: `LayerNorm(x + Sublayer(x))`. The norm wraps around the residual, which destabilizes training at depth because the residual path itself gets renormalized at every block. Pre-norm `x + Sublayer(LayerNorm(x))` leaves the residual path clean and is required for 24+-layer training stability. The Xiong et al. (2020) paper made this canonical. Every modern decoder LLM uses pre-norm. 2. **Sinusoidal absolute positional encoding.** Vaswani 2017 added fixed sinusoidal vectors to the input embedding. This had two problems: it baked position into the input embedding (limiting the model's ability to learn position-aware behavior at later layers), and it generalized poorly past the training context length. **RoPE** (Su et al., 2021) rotates Q and K vectors inside the attention sublayer by a position-dependent angle. The dot product `Q · K^T` then naturally depends on the RELATIVE position between query and key. RoPE generalizes far better to longer contexts (with appropriate base-frequency tuning) and is used by Llama, Mistral, Qwen, DeepSeek, and essentially every other 2026 decoder LLM. 3. **ReLU FFN activation.** Vaswani 2017's FFN was `Linear → ReLU → Linear`. Modern decoder LLMs use **SwiGLU** (Shazeer 2020): `out = Linear((Linear(x)) * Swish(Linear_gate(x)))`. Three matrices and a Swish gate. Pays a roughly 50% parameter cost in the FFN but delivers consistently better quality. Adopted in Llama, Mistral, Qwen, etc. GeGLU is a closely related variant using GeLU instead of Swish. 4. **Full multi-head attention with separate KV per head.** Vaswani 2017 used full MHA: each head has its own K and V projections. For long-context inference, this makes the KV cache enormous. **GQA** (Ainslie et al., 2023) groups multiple query heads to share a single K/V head, shrinking the KV cache by the group size (typically 4x to 8x).

Quick reference

TL;DR

The 2017 Vaswani transformer is a recognizable ancestor of modern LLMs but four of its ingredients were swapped between 2018 and 2024: post-norm became pre-norm, sinusoidal PE became RoPE, ReLU became SwiGLU, and

Memory aid

Easy to grasp

Think of the 2017 transformer like a 2017 sports car. Same number of wheels, same shape of the body, same idea of how the engine sits up front. But over the next ten years they swapped the fuel injection (sinusoidal positions became RoPE), the engine block (ReLU became SwiGLU), where the cooling system sits (post-norm became pre-norm), and how many cylinders share a spark plug (MHA became GQA). The car still looks like a car. The two-seat layout, the chassis, the four-wheel arrangement, those did not change. That is the block recipe. The interview trap option says the chassis got rebuilt; it did not.

Key concepts

Transformer architecture Attention

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: four real swaps (post-norm→pre-norm, sinusoidal→RoPE, ReLU→SwiGLU, MHA→GQA) + the LayerNorm→RMSNorm bonus + identify the trap option (block recipe unchanged) + articulate the 'recipe stable, ingredients evolved' framing.

Side-by-side

Component	Vaswani 2017	Modern 2026 (Llama/Mistral)
Block recipe	Attention + FFN, each residual-wrapped	Attention + FFN, each residual-wrapped (unchanged)
Norm placement	Post-norm	Pre-norm
Norm function	LayerNorm	RMSNorm
Positional encoding	Sinusoidal absolute, added to input	RoPE, applied inside attention
FFN activation	ReLU	SwiGLU
Attention variant	Multi-head attention	Grouped-query attention or MLA
Overall structure	Encoder-decoder	Decoder-only

Real-world examples

Real products, models, and research that use this idea.

Llama 3.1 8B: pre-norm, RoPE (with NTK-aware scaling for 128k context), SwiGLU FFN, GQA (8 KV heads for 32 query heads), RMSNorm. Every Vaswani 2017 ingredient swapped.
Mistral Large 3: pre-norm, RoPE, SwiGLU, GQA, RMSNorm. Same swap inventory.
DeepSeek V4: pre-norm, RoPE, SwiGLU, MLA (instead of GQA), RMSNorm. The MoE replaces some FFN sublayers but the block recipe is otherwise unchanged.
Vaswani 2017 (original): post-norm, sinusoidal PE, ReLU FFN, full MHA, LayerNorm. Every ingredient on the swap list.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf pre-norm is so much better than post-norm at depth, why did Vaswani 2017 use post-norm?

QWhat is the structural difference between RoPE and learned absolute positional embeddings (used in GPT-2 and BERT)?

QWhy has the two-sublayer block recipe survived nine years of architectural innovation?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Including the block recipe itself as 'no longer standard'. It is the single most stable design in deep learning; every modern LLM still uses the attention then FFN two-sublayer template wrapped in residuals.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.