Pick the recipe a 2026 frontier open-weight LLM block actually ships

**Option A is the recipe a 2026 Llama / Mistral / Qwen block actually ships.** The other three each carry at least one obsolete ingredient. Let's break down why Option A is right, by axis. **Norm placement: pre-norm.** Apply the norm BEFORE the attention and FFN sublayers, not after. Pre-norm is what made stacks beyond 20 layers stable to train; every modern LLM uses it. **Norm type: RMSNorm.** Drop the mean-subtraction step and the bias from LayerNorm. About 10-20% cheaper at no measurable quality cost. Llama, Mistral, Qwen, DeepSeek all use RMSNorm. **FFN activation: SwiGLU.** Replace the simple two-matrix FFN with a three-matrix gated variant. The gate (a sigmoid-shaped Swish) lets each channel softly turn on or off. Worth the ~50% extra FFN params at scale. **Attention: GQA.** Multiple query heads share the same K and V heads, shrinking the KV cache that dominates inference memory at long context. Llama 3 70B uses 8 KV heads with 64 query heads, an 8x cache reduction. **Positional: RoPE.** Rotate Q and K vectors by position-dependent angles before the dot product. Gives the attention score a built-in relative-position property without learned positional embeddings and without a hard context cap. **Embeddings: untied.** At modern vocab (128k) and d_model (4k-8k), the unembedding gets its own matrix for capacity. Tying was a small-model trick that costs real quality at scale. Option B is the literal 2017 paper (every dial obsolete). Option C is the 2019 transition (pre-norm landed early; the rest had not). Option D pairs modern ingredients with post-norm, which collapses past about 20 layers.

**Option A: pre-norm + RMSNorm + SwiGLU + GQA + RoPE + untied embeddings.** This is the 2026 consensus block. ### Six axes, one canonical choice each - **Norm placement -> pre-norm.** `x = x + Sublayer(Norm(x))` is what makes 80-layer stacks trainable. Post-norm collapses past ~20 layers. - **Norm type -> RMSNorm.** Drops mean subtraction and bias from LayerNorm. ~10-20% cheaper, equal quality. - **FFN -> SwiGLU.** Three matrices (gate, up, down) with `silu(x @ W_gate) * (x @ W_up)` then projected. ~50% more FFN params than ReLU FFN; the quality lift at fixed compute is consistently positive. - **Attention -> GQA.** Multi-query was too aggressive; full MHA wastes KV cache. GQA with 4-8 KV heads is the sweet spot. DeepSeek's MLA factorizes further but belongs to the same family. - **Positional -> RoPE.** Rotate Q and K by position-dependent angles before the matmul. The dot product becomes a function of (m - n), with no learned PE table and no hard context cap. PI / YaRN extend it past trained context. - **Embeddings -> untied.** Tying saves `vocab_size * d_model` parameters but the quality cost is real at modern scale. ### Why each wrong option fails **Option B** is the unmodified 2017 paper. Post-norm, LayerNorm, ReLU FFN, full MHA, sinusoidal PE, tied embeddings: every axis swapped by 2026. **Option C** is the GPT-2 / Llama-1 transition. Pre-norm landed early, but LayerNorm + GELU + full MHA + learned absolute PE + tied embeddings is the 2019 stack, not 2026. **Option D** is the trap. RMSNorm and GQA look modern, but post-norm is the one ingredient no production lab still ships at scale. The post-norm choice alone disqualifies the recipe.