Match GPT-2 versus Llama-2 attention design choices to their differences

Match pairs·Medium·4.0 · 0·~2 min·Asked atArize Ai Stability Ai·Relevant atMeta

Attempt it

Drag each answer to line up with its matching prompt

Positional encoding

Both use the same scaled dot product attention, softmax(QK^T / sqrt(d_k)) V

Normalization layer

GPT-2 uses GELU vs Llama-2 uses SwiGLU

KV-head sharing (70B scale)

GPT-2 learned absolute PE added at input vs Llama-2 RoPE applied inside attention

Activation in FFN block

GPT-2 plain multi-head attention vs Llama-2 70B grouped query attention (8 KV heads)

Attention math core

GPT-2 uses LayerNorm (mean + variance) vs Llama-2 uses RMSNorm (variance only, no bias)

**GPT-2 and Llama-2 share the same attention math** but differ on almost everything else in the transformer block. The four years between them (2019 to 2023) saw several engineering defaults move, and Llama-2 picked up the better versions. The pieces that changed: - **Positional encoding**: GPT-2 adds a learned absolute PE vector to the token embedding at the input. Llama-2 uses RoPE, which rotates Q and K vectors inside the attention computation so the dot product depends on relative position naturally. - **Normalization**: GPT-2 uses LayerNorm, which centers and rescales using both the mean and variance. Llama-2 uses RMSNorm, which rescales by the root mean square only, with no mean subtraction and no learned bias. - **FFN activation**: GPT-2 uses GELU inside the feed-forward block. Llama-2 uses SwiGLU, a gated variant that combines two linear projections and a SiLU activation. - **KV-head sharing (at 70B)**: GPT-2 has plain multi-head attention with one K and V head per query head. Llama-2 70B uses GQA (grouped-query attention) with 8 KV heads shared across 64 query heads. This gives an 8x KV cache reduction. What stays the same is the attention core itself: `softmax(QK^T/sqrt(d_k)) V`.

**The attention math is identical**. Both compute scaled dot-product attention: $$\text{Attn}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$ Four engineering pieces around it moved between 2019 and 2023. **Positional encoding.** GPT-2 adds a learned absolute PE to the input embedding, so positions 0 through max-len each get a learned vector. The position information enters once at the input and the model has to propagate it through every layer. Llama-2 uses RoPE: it rotates Q and K vectors by position-dependent angles inside the attention computation, so the dot product naturally encodes relative position. RoPE extends to longer contexts more gracefully and doesn't need a learned table. **Normalization.** LayerNorm in GPT-2 subtracts the mean and divides by the standard deviation, then scales and shifts with learned parameters. RMSNorm in Llama-2 drops the mean step (and the bias term), keeping only the RMS divisor. Empirically RMSNorm performs as well as LayerNorm at lower cost. **FFN activation.** GPT-2 uses GELU. Llama-2 uses SwiGLU, a gated linear unit that combines two linear projections with a SiLU activation: `SwiGLU(x) = SiLU(xW_1) * (xW_3) @ W_2`. SwiGLU is more parameter-hungry (three weight matrices instead of two) but consistently improves perplexity per FLOP. **KV-head sharing at 70B.** GPT-2 uses plain multi-head attention. Llama-2 70B uses GQA with 64 query heads grouped onto 8 KV heads, shrinking the KV cache 8x for long-context decode. The central matmul did not move. Everything around it did.

TL;DR

GPT-2 to Llama-2: PE moved to RoPE, LayerNorm became RMSNorm, GELU became SwiGLU, 70B added GQA. The softmax(QK^T/sqrt(d_k))V core stayed identical.

Memory aid

Easy to grasp

Think of two cars from the same family. The 2019 sedan and the 2023 sedan look similar, but the newer one has a smarter steering system, lighter suspension, and a more efficient engine that runs on the same fuel. The seats, the steering wheel, the brakes work the same way. GPT-2 and Llama-2 are like that pair. The core engine, scaled dot-product attention, is unchanged. Around it, every component got swapped for a quieter, faster, or cheaper version: rotary position info instead of added embeddings, simpler normalization, a gated activation, and KV sharing at scale.

Key concepts

RoPE RMSNorm SwiGLU Grouped-query attention

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

GPT-2 (2019) and Llama-2 (2023) are decoder-only transformers with the same overall shape: alternating attention and FFN blocks, residual stream, learned token embeddings, standard scaled dot-product attention. The four-year gap between them was filled with engineering work. Llama-2 swapped four specific components for empirically better versions while keeping the architectural core unchanged.

The pattern matters because it shows what kind of progress was happening. It was not a new attention mechanism or a fundamental rethink. It was careful incremental swaps, each justified by an ablation study.

Positional encoding: learned absolute → RoPE

GPT-2's PE is a learned (max_len, d_model) table added to token embeddings at the input. Each position 0 through max_len gets its own learned vector. The position information enters once at the input layer and the model has to propagate it through every transformer block.

Where the design breaks

There are two failure modes. First, you have no PE vector for positions beyond training-time max_len (hard cap on context length). Second, even within the window, the absolute formulation does not naturally encode the (m - n) shifts that attention scores want.

What RoPE does

RoPE rotates Q and K vectors by position-dependent angles before the attention dot product. The rotation angle at position m is:

\theta_m = m / 10000^{2i/d}

The rotation acts on even-odd pairs of the Q and K dimensions. It has the property that q_m · k_n factors through (m - n). This gives the attention dot product a built-in relative-position property without a learnable table.

Benefits

RoPE has no learnable PE parameters and no hard cap on context (rotations are well-defined at any m). The dot product depends directly on relative position. Long-context extension techniques (RoPE base scaling, NTK-aware, YaRN, LongRoPE) all build on this foundation; a learned absolute PE has no analogous extension story.

Every modern open-weight LLM (Llama 4 Maverick, Qwen 3.5, Mistral Large 3, DeepSeek V4) inherits RoPE. The learned absolute design did not survive the long-context era.

Normalization: LayerNorm → RMSNorm

Activation: GELU → SwiGLU

Head sharing: MHA → GQA at 70B

What stayed the same, and what Llama-2 itself now lacks

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Component	GPT-2 (2019)	Llama-2 (2023)
Positional encoding	Learned absolute PE at input	RoPE applied to Q, K inside attention
Normalization	LayerNorm (mean + variance, with bias)	RMSNorm (variance only, no bias)
FFN activation	GELU	SwiGLU (gated, three weight matrices)
KV-head sharing (70B)	MHA (one KV per query head)	GQA (64 query, 8 KV)
Attention math core	softmax(QK^T/sqrt(d_k)) V	softmax(QK^T/sqrt(d_k)) V (identical)

Real products, models, and research that use this idea.

Llama-2 70B uses GQA with 64 query heads grouped onto 8 KV heads; the 7B and 13B variants keep plain MHA.
Mistral 7B and Mixtral 8x7B inherited Llama-2's stack: RoPE, RMSNorm, SwiGLU, and GQA with 8 KV heads.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does Llama-2 70B specifically pick GQA with 8 KV heads, not 4 or 16?

Meta's Llama 2 ablations compared MHA, MQA, and GQA with 8 KV heads: GQA-8 stayed close to MHA quality, while MQA (1 KV head, the largest possible saving at 64x) lost more. 8 KV heads give an 8x KV cache reduction relative to 64. The GQA paper swept the number of groups and found that 8 groups add little decode cost over MQA. The decision is empirical, anchored to ablation numbers, not derived from theory.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming Llama-2 changed the attention math itself. The core softmax(QK^T/sqrt(d_k))V is identical to GPT-2; what moved is the surrounding engineering: PE, normalization, FFN, head sharing.

60 second bullets to scan on the way to the call.

The four components that changed between GPT-2 and Llama-2
Why RoPE replaced learned absolute PE

Primary sources. Browse if you want the original framing.