Match GPT-2 versus Llama-2 attention design choices to their differences

**GPT-2 and Llama-2 share the same attention math** but differ on almost everything else in the transformer block. The four years between them (2019 to 2023) saw several engineering defaults move, and Llama-2 picked up the better versions. The pieces that changed: - **Positional encoding**: GPT-2 adds a learned absolute PE vector to the token embedding at the input. Llama-2 uses RoPE, which rotates Q and K vectors inside the attention computation so the dot product depends on relative position naturally. - **Normalization**: GPT-2 uses LayerNorm, which centers and rescales using both the running mean and variance. Llama-2 uses RMSNorm, which rescales by the root mean square only, no mean subtraction and no learned bias. - **FFN activation**: GPT-2 uses GELU inside the feed-forward block. Llama-2 uses SwiGLU, a gated variant that combines two linear projections and a SiLU activation. - **KV-head sharing (at 70B)**: GPT-2 has plain multi-head attention with one K and V head per query head. Llama-2 70B uses GQA with 8 KV heads shared across 64 query heads, an 8x KV cache reduction. What stays the same is the attention core itself: `softmax(QK^T/sqrt(d_k)) V`.

**The attention math is identical**, both compute scaled dot-product attention: $$\text{Attn}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$ Four engineering pieces around it moved between 2019 and 2023. **Positional encoding.** GPT-2 adds a learned absolute PE to the input embedding, so positions 0 through max-len each get a learned vector. The position information enters once at the input and the model has to propagate it through every layer. Llama-2 uses RoPE: rotates Q and K vectors by position-dependent angles inside the attention computation, so the dot product naturally encodes relative position. RoPE extends to longer contexts more gracefully and doesn't need a learned table. **Normalization.** LayerNorm in GPT-2 subtracts the running mean and divides by the standard deviation, then scales and shifts with learned parameters. RMSNorm in Llama-2 drops the mean step (and the bias term), keeping only the RMS divisor. Empirically RMSNorm performs as well as LayerNorm at lower cost. **FFN activation.** GPT-2 uses GELU. Llama-2 uses SwiGLU, a gated linear unit that combines two linear projections with a SiLU activation: `SwiGLU(x) = SiLU(xW_1) * (xW_3) @ W_2`. SwiGLU is more parameter-hungry (three weight matrices instead of two) but consistently improves perplexity per FLOP. **KV-head sharing at 70B.** GPT-2 uses plain multi-head attention. Llama-2 70B uses GQA with 64 query heads grouped onto 8 KV heads, shrinking the KV cache 8x for long-context decode. The central matmul did not move. Everything around it did.