**Option A is the recipe a 2026 Llama / Mistral / Qwen block actually ships.** The other three each carry at least one obsolete ingredient.
Let's break down why Option A is right, by axis.
**Norm placement: pre-norm.** Apply the norm BEFORE the attention and FFN sublayers, not after. Pre-norm is what made stacks beyond 20 layers stable to train; every modern LLM uses it.
**Norm type: RMSNorm.** Drop the mean-subtraction step and the bias from LayerNorm. About 10-20% cheaper at no measurable quality cost. Llama, Mistral, Qwen, DeepSeek all use RMSNorm.
**FFN activation: SwiGLU.** Replace the simple two-matrix FFN with a three-matrix gated variant. The gate (a sigmoid-shaped Swish) lets each channel softly turn on or off. Worth the ~50% extra FFN params at scale.
**Attention: GQA.** Multiple query heads share the same K and V heads, shrinking the KV cache that dominates inference memory at long context. Llama 3 70B uses 8 KV heads with 64 query heads, an 8x cache reduction.
**Positional: RoPE.** Rotate Q and K vectors by position-dependent angles before the dot product. Gives the attention score a built-in relative-position property without learned positional embeddings and without a hard context cap.
**Embeddings: untied.** At modern vocab (128k) and d_model (4k-8k), the unembedding gets its own matrix for capacity. Tying was a small-model trick that costs real quality at scale.
Option B is the literal 2017 paper (every dial obsolete). Option C is the 2019 transition (pre-norm landed early; the rest had not). Option D pairs modern ingredients with post-norm, which collapses past about 20 layers.
**Option A: pre-norm + RMSNorm + SwiGLU + GQA + RoPE + untied embeddings.** This is the 2026 consensus block.
### Six axes, one canonical choice each
- **Norm placement -> pre-norm.** `x = x + Sublayer(Norm(x))` is what makes 80-layer stacks trainable. Post-norm collapses past ~20 layers.
- **Norm type -> RMSNorm.** Drops mean subtraction and bias from LayerNorm. ~10-20% cheaper, equal quality.
- **FFN -> SwiGLU.** Three matrices (gate, up, down) with `silu(x @ W_gate) * (x @ W_up)` then projected. ~50% more FFN params than ReLU FFN; the quality lift at fixed compute is consistently positive.
- **Attention -> GQA.** Multi-query was too aggressive; full MHA wastes KV cache. GQA with 4-8 KV heads is the sweet spot. DeepSeek's MLA factorizes further but belongs to the same family.
- **Positional -> RoPE.** Rotate Q and K by position-dependent angles before the matmul. The dot product becomes a function of (m - n), with no learned PE table and no hard context cap. PI / YaRN extend it past trained context.
- **Embeddings -> untied.** Tying saves `vocab_size * d_model` parameters but the quality cost is real at modern scale.
### Why each wrong option fails
**Option B** is the unmodified 2017 paper. Post-norm, LayerNorm, ReLU FFN, full MHA, sinusoidal PE, tied embeddings: every axis swapped by 2026.
**Option C** is the GPT-2 / Llama-1 transition. Pre-norm landed early, but LayerNorm + GELU + full MHA + learned absolute PE + tied embeddings is the 2019 stack, not 2026.
**Option D** is the trap. RMSNorm and GQA look modern, but post-norm is the one ingredient no production lab still ships at scale. The post-norm choice alone disqualifies the recipe.
Quick reference
TL;DR
The 2026 frontier recipe is pre-norm + RMSNorm + SwiGLU + GQA + RoPE + untied embeddings. Every other option mixes at least one obsolete or unstable choice that no production lab still ships.
Imagine six dials on the back of a transformer block. Over eight years the field has nudged each dial one click in the same direction. Norm went from after to before the sublayer. The norm itself dropped its mean-subtraction step to become RMSNorm. The activation gained a gating partner and became SwiGLU. Attention started sharing keys and values across heads to become GQA. Positional information moved from input addition into the attention dot product itself, becoming RoPE. And the output projection became its own table instead of borrowing the input word to numbers phone book. Every modern open-weight frontier LLM ships the exact same combination of dial settings. The wrong answers leave at least one dial in its 2017 or 2018 position, which is enough to disqualify a 2026 production block.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Building a small encoder-only model for retrieval embeddings Encoder-only models like ModernBERT use a different recipe (bidirectional attention, MLM objective). GQA and RoPE are less critical when there is no autoregressive decode loop.
Reproducing a published 2017-2019 baseline for a research paper If the paper used post-norm + LayerNorm + GELU, you must match exactly to compare apples to apples. Do not silently modernize the baseline.
Training a tiny model (under 100M parameters) Tied embeddings save a meaningful fraction of parameters at small scale. The trade flips around 1B params; under that, tying is defensible.
Targeting an inference stack that does not support RoPE kernels Some embedded or specialized inference runtimes lack fused RoPE. ALiBi or learned absolute PE may be the pragmatic choice if you cannot extend the runtime.
Quick scan · 2–4 min
Everything important, quickly.
5 min: walk all six axes, explain why each modern choice won its ablation race, name a 2026 model that ships each choice, and call out why Option D's post-norm placement is the disqualifying ingredient.
Side-by-side
Axis
2017 paper (Option B)
2019 transition (Option C)
2026 consensus (Option A)
Norm placement
Post-norm
Pre-norm
Pre-norm
Norm type
LayerNorm
LayerNorm
RMSNorm
FFN activation
ReLU
GELU
SwiGLU
Attention variant
Full MHA
Full MHA
GQA (or MLA)
Positional encoding
Sinusoidal
Learned absolute
RoPE
Embedding tying
Tied
Tied
Untied
Real-world examples
Real products, models, and research that use this idea.
2Mistral Large 3 (Mistral, 2025): same six axes; GQA with sliding-window attention as an additional axis on top.
3Qwen 3.5 (Alibaba, 2025-2026): identical recipe; varies number of layers and head counts but every axis matches.
4DeepSeek V4 (DeepSeek, 2026): substitutes MLA (multi-head latent attention) for GQA (same family idea: share KV across heads, even more aggressive) and adds MoE blocks; the other five axes are unchanged.
5Llama 3.1 405B: same six choices, scaled to 126 layers and d_model=16384. Same recipe, larger dials.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy did the field converge on this exact six-axis recipe across so many independent labs?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the next swap likely to be on this block?
Sign in or go Pro to see the approach for this follow-up.
QIf you had to drop one axis modernization for compute reasons, which would you keep last?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Picking Option D because it has RMSNorm and GQA. The post-norm placement disqualifies it immediately; no 2026 frontier LLM ships post-norm at the depths these models reach.