Tying reuses the [128k, 2048] embedding matrix as the output head, saving the entire 262M parameter unembedding tensor.
Imagine your model has two big phone books: one to look up the meaning of each word at the input, and one to score every possible next word at the output. Both books have a row per word in the vocabulary and the same number of columns. They are not the same book, they just have the exact same shape. Weight tying says: drop the second book, just flip the first one sideways and use it for output scoring too. With a vocabulary of 128,000 words and 2048-wide rows, the dropped book held about 262 million numbers. For a small model that is a huge chunk of total weights, which is why small Llamas and Gemmas tie the two books while giant models keep them separate.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Weight tying is one of those design choices that looks like a thrifty implementation trick and turns out to be a defensible architectural decision worth tens of millions of parameters. The question 'should we tie the input and output embeddings' has a wrong answer (never tie) only in narrow cases; for most small models, tying is the right call by a clear margin, and the parameter saving is exactly the size of the unembedding matrix.
This walkthrough does three things. First, it grounds the arithmetic so 262M stops being a number you have to memorise. Second, it explains why the design choice is genuinely contested at the frontier, where the savings are small but the quality tradeoff matters. Third, it covers the implementation trap that bites a third of engineers who try to tie embeddings for the first time.
Mental model: the input embedding is
[V, d]. The untied output head is also[V, d]. They have the same shape. Tying says: stop pretending they are different tensors and just store one.
The arithmetic and why this matters at small scale
At V = 128000 and d_model = 2048, the unembedding head is 128000 x 2048 = 262,144,000 parameters, about 262M. Tying saves exactly this tensor.
Why the saving matters most for small models
Look at the parameter fraction. A 1B model with a 128k vocab has roughly (128k x 2048) x 2 = 524M parameters in its embedding ends if untied, more than half the model. Tying cuts that to 262M, leaving the other 738M for the actual transformer stack. Without tying, a 1B 128k-vocab model would have less compute capacity than a 500M 32k-vocab model.
This is exactly why Llama 3.2 1B and 3B ship with tied embeddings, why Gemma 2 2B ties, and why every published mobile or edge LLM ties. The vocabulary stayed at frontier scale (128k for multilingual coverage), but the transformer stack shrank, so the embedding fraction ballooned.
Why the saving matters less at scale
A 70B model has the same 262M embedding-end tensor (assuming the same vocab and d_model order of magnitude), but as a fraction it is under 0.4%. At that point the question is no longer 'can we afford the parameters' but 'does sharing the input and output geometries help or hurt'. Empirically, large models gain slightly from untying because the output head learns its own logit-calibration geometry, distinct from the input recognition geometry. That gain is small but consistent.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3.2 1B and 3B tie input and output embeddings, since the 128k vocab embedding head would otherwise be a third of the model.
- Llama 3 70B and Llama 4 untie, preferring the extra 262M parameters of a separate output head for the perplexity it buys.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does tying interact with the input layernorm and final layernorm?
Tying ties the matrices but does not tie the surrounding norms; a final RMSNorm before the head can effectively rescale the output side independently of the input.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing tying with quantization or activation sharing. Tying drops a parameter tensor entirely; it has nothing to do with bias, activations, or sharing FFN weights across layers.
60 second bullets to scan on the way to the call.
Shape of the embedding matrix and the untied output head
How to compute the parameter saving from tying
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.