Tied embeddings share one weight matrix between the input token embedding lookup and the output unembedding / logits projection. The output head reuses the input matrix transposed, saving vocab*d_model parameters. GPT-2 and Llama-1 used this. Llama-3+ untie them because at >7B scale the input embedding and output head learn different distributions (input is lookup by id, output is similarity to vocab) and the small quality bump from a dedicated output matrix is worth the extra ~500M parameters.
**Tied embeddings share one matrix for input and output.** A transformer LM has two places where vocab meets hidden dimension. At the input, the token embedding lookup maps a token id to a `d_model`-dimensional vector (shape `(vocab, d_model)`). At the output, the unembedding head projects the final hidden vector back to a distribution over the vocab (shape `(d_model, vocab)`).
Weight tying says: use the SAME matrix for both. The output head is just the transpose of the input embedding. `logits = final_hidden @ W_in^T`. This saves one `(vocab, d_model)` matrix worth of parameters, which is significant when vocab is 128k and d_model is 4096 (~525M params at Llama-3 sizes).
**Why GPT-2 and Llama-1 tied.** At small scale, the parameter savings mattered, and a body of research (Press and Wolf, 2017) had shown tying helped or was neutral.
**Why Llama-3 and successors untied.** At 8B+ scale, the parameter cost of untying is small as a fraction of the total. And empirically the dedicated output matrix learns a slightly different representation than the input embedding, which delivers a small but consistent quality lift on standard benchmarks. The trade is: ~525M extra params for ~0.5-1 point of quality. At frontier scale, that trade is worth it; at toy scale, it is not.
**The mechanics of weight tying.** A transformer LM has two `(vocab, d_model)` weight matrices: `W_in` for the input token embedding lookup, and `W_out` for the output unembedding head that produces logits. Weight tying enforces `W_out = W_in^T`, so the SAME `(vocab, d_model)` matrix is reused at both ends of the network. The output computation becomes `logits = final_hidden @ W_in^T`.
The parameter savings are exactly `vocab * d_model`. At Llama-3 8B sizes (vocab 128k, d_model 4096), that is about 525M parameters, or 6-7 percent of the 8B model.
**The history.** Press and Wolf (2017) introduced weight tying with strong evidence that it improved perplexity at the scales of the day (sub-1B). GPT-2 and most small to mid-scale transformer LMs adopted it. Llama-1 (2023) tied embeddings for both 7B and 13B sizes. Mistral 7B (2023) also tied.
**Why modern frontier models untie.** Llama-2 70B untied; Llama-3 untied across all sizes; DeepSeek V2 and V3 untied; Qwen 2 untied at the larger sizes. The shift is empirical: at frontier scale, the input embedding and the output head learn different statistical roles. The input embedding is fundamentally a lookup table indexed by token id: row `i` of `W_in` is just 'what does token i mean'. The output head is a similarity scorer: it takes a hidden vector and produces a score for every vocab item. The two jobs need different representations, and forcing them to share one matrix costs measurable quality at scale (typically 0.5-1 point on standard LM benchmarks).
The parameter cost is real but small relative to total model size. At 8B, 525M is ~6.5 percent. At 70B, it is under 1 percent. The trade is favorable for frontier-scale models where every fraction of a benchmark point is fought over.
**Quick rule.** Below ~3B, tie. From ~7B onward, untie. The 2026 production default for decoder LLMs is untied.
Quick reference
TL;DR
Weight tying shares one `(vocab, d_model)` matrix between the input embedding lookup and the output logits projection (`W_out = W_in^T`). Saves vocab*d_model params (~525M at Llama-3 8B sizes).
Imagine a dictionary you use both to look up words by id (input) and to score how well a sentence ends with each word (output). Tying says: one dictionary, use it both ways. Untying says: keep two dictionaries, one optimized for lookup and one optimized for scoring. At small scale, one dictionary is fine. At frontier scale, having two specialists each doing their own job consistently scores a fraction of a point higher on benchmarks. Llama-3 and most >7B production models pay the extra storage to get the lift.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Training a sub-3B model where parameter efficiency dominates At small scale, the embedding table is 15-20% of the model and untying is hard to justify; the quality lift does not pay for the storage.
Fine-tuning an existing model The base model's tying choice is baked into its weights; switching mid-training is impractical and risks training instability.
Inference stacks that offload the embedding table When the embedding table sits on a different device or quantization tier (CPU embed + GPU compute, or INT8 embed + BF16 head), tying forces input and output to share placement and precision, which often costs more than untying saves.
Quick scan · 2–4 min
Everything important, quickly.
2 min: define tying (shared `W_in` matrix used as `W_in^T` at output), state the param cost (`vocab * d_model`, ~525M at Llama-3 8B), give the modern default (untie at >7B, tie below ~3B), and explain WHY (input is a lookup, output is a scorer; at scale they want different representations).
Side-by-side
Aspect
Tied embeddings
Untied embeddings
Param count at vocab=128k, d=4096
Shared (saves ~525M)
Two matrices (costs ~525M extra)
Input embedding role
Lookup AND scoring (compromise)
Pure lookup
Output head role
Scoring AND lookup (compromise)
Pure similarity scorer
Quality at >7B scale
Slight regression
Slight lift (~0.5-1 benchmark point)
Quality at <3B scale
Roughly equal or better (param efficiency)
Slightly worse per-param
2026 frontier convention
Used in small LMs (<3B)
Default for 7B+ decoder LLMs
Real-world examples
Real products, models, and research that use this idea.
1Llama-3 8B, 70B, and 405B all untie embeddings. The 8B pays ~525M extra params for the dedicated output head.
2DeepSeek V3 (671B MoE) unties its embedding from its output head, consistent with the frontier convention.
3Llama-1 7B and 13B (2023) tied embeddings; Llama-2 7B and 13B tied; Llama-2 70B untied; Llama-3 untied across all sizes. The transition tracks scale crossing the 70B threshold.
4Llama-3.2 1B and 3B (small variants) tied embeddings, consistent with the 'tie below ~3B' rule of thumb.
5Press and Wolf (2017) introduced weight tying in the era of <1B models and demonstrated it improved perplexity at that scale.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QIf untying gives the output head freedom to specialize, why not also use a bigger d_model JUST at the output head?
Sign in or go Pro to see the approach for this follow-up.
QWhat happens at training time if you accidentally tie embeddings when the codebase expects untied?
Sign in or go Pro to see the approach for this follow-up.
QHow does weight tying interact with vocabulary expansion (adding new tokens to a tied model)?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming tied embeddings are always the right default. They were for GPT-2 and Llama-1, but every major decoder LLM from Llama-3 onward untied them deliberately.