Defend untying the output head: what does an extra ~500M parameters at the tail of the stack buy?

**The team would save ~525M params, but lose a small quality lift.** The input embedding (shape `(128k, 4096)` for Llama-3 8B) and the output head (shape `(4096, 128k)`) are two matrices that together cost about 1.05B parameters. Tying them shares one matrix between both jobs, saving ~525M. **What they lose by tying.** The input embedding's job is a lookup: take a token id, return a 4096-dimensional vector that encodes 'what does this token mean'. The output head's job is similarity scoring: take the final residual stream vector and return a logit for every vocab item. These are mathematically dual operations but they receive very different training signals. The input embedding row for token `i` is updated only when token `i` appears in the input batch. The output head row for token `i` is updated on every prediction step, because the cross-entropy loss touches every vocab logit. With tying, one matrix has to satisfy both gradient flows simultaneously. With untying, each matrix specializes. At 8B scale, that specialization is worth roughly 0.5-1 point on standard benchmarks. **Why Llama-3 untied.** Meta's technical report notes that untying contributed to the quality gap between Llama-2 and Llama-3. The 525M parameter cost is ~6.5 percent of an 8B model: small enough to absorb in exchange for the quality lift. Llama-3 untied across all sizes (8B, 70B, 405B), and most frontier 2026 decoder LLMs (DeepSeek V3, Qwen 2 large, Mistral Large) made the same choice. **Quick rule.** Below ~3B, tie (parameter ratio too unfavorable to untie). From 7B onward, untie. The 8B-team in the question should untie.

**The trade in one sentence.** Tying saves `vocab * d_model` parameters (~525M for Llama-3 8B sizes) but forces one matrix to serve two operations with structurally different gradient signals; at 8B+ scale the small quality lift from untying (0.5-1 benchmark point) outweighs the parameter savings. **Why the input embedding and output head are statistically different.** The input embedding `W_in` of shape `(vocab, d_model)` is indexed by token id. Row `i` is the embedding for token `i` and is only touched in the forward pass when token `i` appears in the input. Its gradient is structurally sparse: rare tokens get rare updates, common tokens get frequent updates. The signal pushes each row toward 'a useful starting representation for the model to refine across all layers'. The output head `W_out` of shape `(d_model, vocab)` is multiplied against the final hidden state to produce all vocab logits at once. Column `i` (or row `i` of `W_out^T`) is the 'next-token preference vector' for token `i`. Its gradient is dense: cross-entropy loss against the predicted distribution updates every column on every step, even columns whose tokens did not appear. The signal pushes each column toward being a good similarity scorer against the typical residual stream direction associated with predicting that token. **Why the compromise hurts at scale.** Tying forces these two distinct optimization signals onto one matrix. At small scale (sub-3B), the matrix is large enough relative to model size that the parameter savings dominate and the compromise is tolerable. At 7B+, the parameter ratio shrinks (from ~15-20 percent of model to ~6-7 percent and below), and the cost of the compromise becomes visible: roughly 0.5-1 point on MMLU, HumanEval, and similar standard benchmarks. **Cost accounting.** Untying costs exactly `vocab * d_model` parameters: 524M for Llama-3 8B, 1.05B for 70B, 2.1B for 405B. The ratio shrinks as the model grows: 6.5% of 8B, 1.5% of 70B, 0.5% of 405B.