Pick the recipe a 2026 frontier open weight LLM block actually ships

MCQ·Medium·4.0 · 0·~1 min·Asked atFigure Ai Mercor Persistent·Relevant atAnthropic Databricks Meta Mistral AI

Attempt it

**Option A is the recipe a 2026 Llama / Mistral / Qwen block actually ships.** The other three each carry at least one obsolete ingredient. Let's break down why Option A is right, by axis. **Norm placement: pre-norm.** Apply the norm BEFORE the attention and FFN sublayers, not after. Pre-norm is what made stacks beyond 20 layers stable to train; every modern LLM uses it. **Norm type: RMSNorm.** Drop the mean-subtraction step and the bias from LayerNorm. About 10-20% cheaper at no measurable quality cost. Llama, Mistral, Qwen, DeepSeek all use RMSNorm. **FFN activation: SwiGLU.** Replace the simple two-matrix FFN with a three-matrix gated variant. The gate (a sigmoid-shaped Swish) lets each channel softly turn on or off. Worth the ~50% extra FFN params at scale. **Attention: GQA.** Multiple query heads share the same K and V heads, shrinking the KV cache that dominates inference memory at long context. Llama 3 70B uses 8 KV heads with 64 query heads, an 8x cache reduction. **Positional: RoPE.** Rotate Q and K vectors by position-dependent angles before the dot product. Gives the attention score a built-in relative-position property without learned positional embeddings and without a hard context cap. **Embeddings: untied.** At modern vocab (128k) and d_model (4k-8k), the unembedding gets its own matrix for capacity. Tying was a small-model trick that costs real quality at scale. Option B is the literal 2017 paper (every dial obsolete). Option C is the 2019 transition (pre-norm landed early; the rest had not). Option D pairs modern ingredients with post-norm, which collapses past about 20 layers.

**Option A: pre-norm + RMSNorm + SwiGLU + GQA + RoPE + untied embeddings.** This is the 2026 consensus block. ### Six axes, one canonical choice each - **Norm placement -> pre-norm.** `x = x + Sublayer(Norm(x))` is what makes 80-layer stacks trainable. Post-norm collapses past ~20 layers. - **Norm type -> RMSNorm.** Drops mean subtraction and bias from LayerNorm. ~10-20% cheaper, equal quality. - **FFN -> SwiGLU.** Three matrices (gate, up, down) with `silu(x @ W_gate) * (x @ W_up)` then projected. ~50% more FFN params than ReLU FFN; the quality lift at fixed compute is consistently positive. - **Attention -> GQA.** Multi-query was too aggressive; full MHA wastes KV cache. GQA with 4-8 KV heads is the sweet spot. DeepSeek's MLA factorizes further but belongs to the same family. - **Positional -> RoPE.** Rotate Q and K by position-dependent angles before the matmul. The dot product becomes a function of (m - n), with no learned PE table and no hard context cap. PI / YaRN extend it past trained context. - **Embeddings -> untied.** Tying saves `vocab_size * d_model` parameters but the quality cost is real at modern scale. ### Why each wrong option fails **Option B** is the unmodified 2017 paper. Post-norm, LayerNorm, ReLU FFN, full MHA, sinusoidal PE, tied embeddings: every axis swapped by 2026. **Option C** is the GPT-2 / Llama-1 transition. Pre-norm landed early, but LayerNorm + GELU + full MHA + learned absolute PE + tied embeddings is the 2019 stack, not 2026. **Option D** is the trap. RMSNorm and GQA look modern, but post-norm is the one ingredient no production lab still ships at scale. The post-norm choice alone disqualifies the recipe.

TL;DR

The 2026 frontier recipe is pre-norm + RMSNorm + SwiGLU + GQA + RoPE + untied embeddings. Every other option mixes at least one obsolete or unstable choice that no production lab still ships.

Memory aid

Easy to grasp

Imagine six dials on the back of a transformer block. Over eight years the field has nudged each dial one click in the same direction. Norm went from after to before the sublayer. The norm itself dropped its mean-subtraction step to become RMSNorm. The activation gained a gating partner and became SwiGLU. Attention started sharing keys and values across heads to become GQA. Positional information moved from input addition into the attention dot product itself, becoming RoPE. And the output projection became its own table instead of borrowing the input word to numbers phone book. Every modern open-weight frontier LLM ships the exact same combination of dial settings. The wrong answers leave at least one dial in its 2017 or 2018 position, which is enough to disqualify a 2026 production block.

Key concepts

RMSNorm SwiGLU activation Grouped-query attention Rotary positional embeddings

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

If you have to pick which transformer block recipe a 2026 frontier open-weight model ships, you are really being asked whether you can read a config file. The convergence across labs is so tight that one set of six choices appears in essentially every Llama, Mistral, Qwen, and DeepSeek config you will encounter in production.

This card walks the six axes, gives each modern choice its history in two paragraphs, names a 2026 model that ships it, and explains specifically why the three wrong options each fail at least one axis hard enough to disqualify the recipe.

Axis 1-2: norm placement and norm type

Pre-norm placement. The original 2017 architecture used post-norm: x = LayerNorm(x + Sublayer(x)). The norm sits AFTER the residual sum. This was the Vaswani default and worked for the 6-layer encoder-decoder of the original Attention Is All You Need. It does not scale.

Around 2019-2020 (On Layer Normalization in the Transformer Architecture, Xiong et al.), the field discovered that swapping to pre-norm (x = x + Sublayer(LayerNorm(x)), with the norm on the sublayer's INPUT) produces gradient flow that does not require heavy warmup and trains 30-100 layer stacks stably. Every modern frontier LLM uses pre-norm. The cost is that the residual stream grows un-normalized through the stack, which is why pre-norm models add a final RMSNorm before the LM head.

RMSNorm. LayerNorm: subtract the mean across d_model, divide by the sqrt of variance, scale by a learned per-channel gamma, add a learned per-channel beta. RMSNorm: just divide by sqrt of mean-square along d_model, scale by gamma. Two operations dropped (mean subtraction, beta bias) and the kernel is faster.

The Llama lineage popularized RMSNorm in 2023. Ablations show it matches LayerNorm in final quality and saves ~10-20% on the norm kernel cost. Llama 3, Mistral, Qwen, DeepSeek all use RMSNorm everywhere LayerNorm would have gone. This is one of the cheapest wins in the modern recipe: a strict Pareto improvement.

Axis 3: FFN activation (SwiGLU)

Axis 4-5: attention variant and positional encoding

Axis 6 and why each wrong option fails

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Axis	2017 paper (Option B)	2019 transition (Option C)	2026 consensus (Option A)
Norm placement	Post-norm	Pre-norm	Pre-norm
Norm type	LayerNorm	LayerNorm	RMSNorm
FFN activation	ReLU	GELU	SwiGLU
Attention variant	Full MHA	Full MHA	GQA (or MLA)
Positional encoding	Sinusoidal	Learned absolute	RoPE
Embedding tying	Tied	Tied	Untied

Real products, models, and research that use this idea.

Llama 3 70B (Meta, 2024): pre-norm + RMSNorm + SwiGLU + GQA (64 query heads, 8 KV heads) + RoPE + untied 128k-vocab embeddings.
Mistral Large 3 (Mistral, 2025): same six axes; GQA with sliding-window attention as an additional axis on top.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy did the field converge on this exact six-axis recipe across so many independent labs?

Each swap was validated by a paper with strong ablations (pre-norm: 2019; RMSNorm: 2019; SwiGLU: 2020 GLU Variants; GQA: 2023; RoPE: 2021 RoFormer). Once Llama 1/2 shipped the combined recipe and trained well, subsequent labs copied because the cost of deviating without a clear quality story is high. Empirical convergence under competitive pressure, not coordination.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Picking Option D because it has RMSNorm and GQA. The post-norm placement disqualifies it immediately; no 2026 frontier LLM ships post-norm at the depths these models reach.

60 second bullets to scan on the way to the call.

Six axes of a modern transformer block and the canonical choice on each
Pre-norm vs post-norm: which one trains 80-layer stacks stably

Primary sources. Browse if you want the original framing.