Define tied embeddings and explain why Llama-3 chose to untie them

Flashcard·Medium·4.0 · 0·~30s·Relevant atAnthropic Mistral AI

Attempt it

Tied embeddings share one weight matrix between the input token embedding lookup and the output unembedding / logits projection. The output head reuses the input matrix transposed, saving vocab*d_model parameters. GPT-2 and Llama-1 used this. Llama-3+ untie them because at >7B scale the input embedding and output head learn different distributions (input is lookup by id, output is similarity to vocab) and the small quality bump from a dedicated output matrix is worth the extra ~500M parameters.

Your answer

**Tied embeddings share one matrix for input and output.** A transformer LM has two places where vocab meets hidden dimension. At the input, the token embedding lookup maps a token id to a `d_model`-dimensional vector (shape `(vocab, d_model)`). At the output, the unembedding head projects the final hidden vector back to a distribution over the vocab (shape `(d_model, vocab)`). Weight tying says: use the SAME matrix for both. The output head is just the transpose of the input embedding. `logits = final_hidden @ W_in^T`. This saves one `(vocab, d_model)` matrix worth of parameters, which is significant when vocab is 128k and d_model is 4096 (~525M params at Llama-3 sizes). **Why GPT-2 and Llama-1 tied.** At small scale, the parameter savings mattered, and a body of research (Press and Wolf, 2017) had shown tying helped or was neutral. **Why Llama-3 and successors untied.** At 8B+ scale, the parameter cost of untying is small as a fraction of the total. And empirically the dedicated output matrix learns a slightly different representation than the input embedding, which delivers a small but consistent quality lift on standard benchmarks. The trade is: ~525M extra params for ~0.5-1 point of quality. At frontier scale, that trade is worth it; at toy scale, it is not.

**The mechanics of weight tying.** A transformer LM has two `(vocab, d_model)` weight matrices: `W_in` for the input token embedding lookup, and `W_out` for the output unembedding head that produces logits. Weight tying enforces `W_out = W_in^T`, so the SAME `(vocab, d_model)` matrix is reused at both ends of the network. The output computation becomes `logits = final_hidden @ W_in^T`. The parameter savings are exactly `vocab * d_model`. At Llama-3 8B sizes (vocab 128k, d_model 4096), that is about 525M parameters, or 6-7 percent of the 8B model. **The history.** Press and Wolf (2017) introduced weight tying with strong evidence that it improved perplexity at the scales of the day (sub-1B). GPT-2 and most small to mid-scale transformer LMs adopted it. Llama-1 (2023) tied embeddings for both 7B and 13B sizes. Mistral 7B (2023) also tied. **Why modern frontier models untie.** Llama-2 70B untied; Llama-3 untied across all sizes; DeepSeek V2 and V3 untied; Qwen 2 untied at the larger sizes. The shift is empirical: at frontier scale, the input embedding and the output head learn different statistical roles. The input embedding is fundamentally a lookup table indexed by token id: row `i` of `W_in` is just 'what does token i mean'. The output head is a similarity scorer: it takes a hidden vector and produces a score for every vocab item. The two jobs need different representations, and forcing them to share one matrix costs measurable quality at scale (typically 0.5-1 point on standard LM benchmarks). The parameter cost is real but small relative to total model size. At 8B, 525M is ~6.5 percent. At 70B, it is under 1 percent. The trade is favorable for frontier-scale models where every fraction of a benchmark point is fought over. **Quick rule.** Below ~3B, tie. From ~7B onward, untie. The 2026 production default for decoder LLMs is untied.

Quick reference

TL;DR

Weight tying shares one `(vocab, d_model)` matrix between the input embedding lookup and the output logits projection (`W_out = W_in^T`). Saves vocab*d_model params (~525M at Llama-3 8B sizes).

Memory aid

Easy to grasp

Imagine a dictionary you use both to look up words by id (input) and to score how well a sentence ends with each word (output). Tying says: one dictionary, use it both ways. Untying says: keep two dictionaries, one optimized for lookup and one optimized for scoring. At small scale, one dictionary is fine. At frontier scale, having two specialists each doing their own job consistently scores a fraction of a point higher on benchmarks. Llama-3 and most >7B production models pay the extra storage to get the lift.

Key concepts

Transformer architecture

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

2 min: define tying (shared `W_in` matrix used as `W_in^T` at output), state the param cost (`vocab * d_model`, ~525M at Llama-3 8B), give the modern default (untie at >7B, tie below ~3B), and explain WHY (input is a lookup, output is a scorer; at scale they want different representations).

Side-by-side

Aspect	Tied embeddings	Untied embeddings
Param count at vocab=128k, d=4096	Shared (saves ~525M)	Two matrices (costs ~525M extra)
Input embedding role	Lookup AND scoring (compromise)	Pure lookup
Output head role	Scoring AND lookup (compromise)	Pure similarity scorer
Quality at >7B scale	Slight regression	Slight lift (~0.5-1 benchmark point)
Quality at <3B scale	Roughly equal or better (param efficiency)	Slightly worse per-param
2026 frontier convention	Used in small LMs (<3B)	Default for 7B+ decoder LLMs

Real-world examples

Real products, models, and research that use this idea.

Llama-3 8B, 70B, and 405B all untie embeddings. The 8B pays ~525M extra params for the dedicated output head.
DeepSeek V3 (671B MoE) unties its embedding from its output head, consistent with the frontier convention.
Llama-1 7B and 13B (2023) tied embeddings; Llama-2 7B and 13B tied; Llama-2 70B untied; Llama-3 untied across all sizes. The transition tracks scale crossing the 70B threshold.
Llama-3.2 1B and 3B (small variants) tied embeddings, consistent with the 'tie below ~3B' rule of thumb.
Press and Wolf (2017) introduced weight tying in the era of <1B models and demonstrated it improved perplexity at that scale.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf untying gives the output head freedom to specialize, why not also use a bigger d_model JUST at the output head?

QWhat happens at training time if you accidentally tie embeddings when the codebase expects untied?

QHow does weight tying interact with vocabulary expansion (adding new tokens to a tied model)?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming tied embeddings are always the right default. They were for GPT-2 and Llama-1, but every major decoder LLM from Llama-3 onward untied them deliberately.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.