Define tied embeddings and explain why Llama-3 chose to untie them

Flashcard·Medium·4.0 · 0·~30s·Asked atAccenture Baseten Lambda Labs·Relevant atAnthropic Mistral AI

Attempt it

Tied embeddings share one weight matrix between the input token embedding lookup and the output unembedding / logits projection. The output head reuses the input matrix transposed, saving vocab*d_model parameters. GPT-2 and Llama-1 used this. Llama-3+ untie them because at >7B scale the input embedding and output head learn different distributions (input is lookup by id, output is similarity to vocab) and the small quality bump from a dedicated output matrix is worth the extra ~500M parameters.

Your answer

**Tied embeddings share one matrix for input and output.** A transformer LM has two places where vocab meets hidden dimension. At the input, the token embedding lookup maps a token id to a `d_model`-dimensional vector (shape `(vocab, d_model)`). At the output, the unembedding head projects the final hidden vector back to a distribution over the vocab (shape `(d_model, vocab)`). Weight tying says: use the SAME matrix for both. The output head is just the transpose of the input embedding. `logits = final_hidden @ W_in^T`. This saves one `(vocab, d_model)` matrix worth of parameters, which is significant when vocab is 128k and d_model is 4096 (~525M params at Llama-3 sizes). **Why GPT-2 and Llama-1 tied.** At small scale, the parameter savings mattered, and a body of research (Press and Wolf, 2017) had shown tying helped or was neutral. **Why Llama-3 and successors untied.** At 8B+ scale, the parameter cost of untying is small as a fraction of the total. And empirically the dedicated output matrix learns a slightly different representation than the input embedding, which delivers a small but consistent quality lift on standard benchmarks. The trade is: ~525M extra params for ~0.5-1 point of quality. At frontier scale, that trade is worth it; at toy scale, it is not.

**The mechanics of weight tying.** A transformer LM has two `(vocab, d_model)` weight matrices: `W_in` for the input token embedding lookup, and `W_out` for the output unembedding head that produces logits. Weight tying enforces `W_out = W_in^T`, so the SAME `(vocab, d_model)` matrix is reused at both ends of the network. The output computation becomes `logits = final_hidden @ W_in^T`. The parameter savings are exactly `vocab * d_model`. At Llama-3 8B sizes (vocab 128k, d_model 4096), that is about 525M parameters, or 6-7 percent of the 8B model. **The history.** Press and Wolf (2017) introduced weight tying with strong evidence that it improved perplexity at the scales of the day (sub-1B). GPT-2 and most small to mid-scale transformer LMs adopted it. Llama-1 (2023) tied embeddings for both 7B and 13B sizes. Mistral 7B (2023) also tied. **Why modern frontier models untie.** Llama-2 70B untied; Llama-3 untied across all sizes; DeepSeek V2 and V3 untied; Qwen 2 untied at the larger sizes. The shift is empirical: at frontier scale, the input embedding and the output head learn different statistical roles. The input embedding is fundamentally a lookup table indexed by token id: row `i` of `W_in` is just 'what does token i mean'. The output head is a similarity scorer: it takes a hidden vector and produces a score for every vocab item. The two jobs need different representations, and forcing them to share one matrix costs measurable quality at scale (typically 0.5-1 point on standard LM benchmarks). The parameter cost is real but small relative to total model size. At 8B, 525M is ~6.5 percent. At 70B, it is under 1 percent. The trade is favorable for frontier-scale models where every fraction of a benchmark point is fought over. **Quick rule.** Below ~3B, tie. From ~7B onward, untie. The 2026 production default for decoder LLMs is untied.

TL;DR

Weight tying shares one `(vocab, d_model)` matrix between the input embedding lookup and the output logits projection (`W_out = W_in^T`). Saves vocab*d_model params (~525M at Llama-3 8B sizes).

Memory aid

Easy to grasp

Imagine a dictionary you use both to look up words by id (input) and to score how well a sentence ends with each word (output). Tying says: one dictionary, use it both ways. Untying says: keep two dictionaries, one optimized for lookup and one optimized for scoring. At small scale, one dictionary is fine. At frontier scale, having two specialists each doing their own job consistently scores a fraction of a point higher on benchmarks. Llama-3 and most >7B production models pay the extra storage to get the lift.

Key concepts

Transformer architecture

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Weight tying is one of those design decisions that looks tiny on paper but has been deliberately flipped between every major generation of frontier LLMs. The interview question is whether you can articulate both the mechanics (what is shared and what it saves) and the empirical reason the convention shifted.

The two embedding matrices in a transformer LM

Every transformer LM has two vocab by width weight matrices that bridge tokens and hidden dimensions.

Input embedding. Shape vocab by model width. Indexed by token id; row i is the embedding vector for token i. The input forward pass is a pure lookup that pulls one row out of this table.

Output unembedding (LM head). Shape model width by vocab. Multiplied by the final hidden state to emit logits. Each column is a width-dimensional 'next-token preference vector' for one vocab item. The dot product of the residual stream with that vector is the logit for that token.

Both matrices have the same vocab by width shape (modulo transpose). They sit at opposite ends of the network. Weight tying says: use the SAME matrix for both, so the output head is just the transpose of the input embedding.

What tying saves and what it gives up

Why the two operations want different representations

What 2026 frontier models do

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Aspect	Tied embeddings	Untied embeddings
Param count at vocab=128k, d=4096	Shared (saves ~525M)	Two matrices (costs ~525M extra)
Input embedding role	Lookup AND scoring (compromise)	Pure lookup
Output head role	Scoring AND lookup (compromise)	Pure similarity scorer
Quality at >7B scale	Slight regression	Slight lift (~0.5-1 benchmark point)
Quality at <3B scale	Roughly equal or better (param efficiency)	Slightly worse per-param
2026 frontier convention	Used in small LMs (<3B)	Default for 7B+ decoder LLMs

Real products, models, and research that use this idea.

Llama-3 8B, 70B, and 405B all untie embeddings. The 8B pays ~525M extra params for the dedicated output head.
DeepSeek V3 (671B MoE) unties its embedding from its output head, consistent with the frontier convention.

What an interviewer would ask next. Try answering before peeking at the approach.

QIf untying gives the output head freedom to specialize, why not also use a bigger d_model JUST at the output head?

Because the output head's input is the final residual stream, which has dimension d_model. You cannot widen one without widening the whole stack. Some designs add a final projection layer (d_model -> d_out -> vocab) but the extra parameters could equivalently be spent on widening or deepening, which usually pays off more.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming tied embeddings are always the right default. They were for GPT-2 and Llama-1, but every major decoder LLM from Llama-3 onward untied them deliberately.

60 second bullets to scan on the way to the call.

What weight tying actually shares (one vocab by width matrix between input and output)
Parameter cost of untying: exactly the vocab by width product

Primary sources. Browse if you want the original framing.