Walk through what quadrupling vocab to 128k does to params, compute, and sequences
Your team retrains an 8B model's tokenizer, growing the vocabulary from 32k to 128k. Walk through the architectural consequences: parameter count, the softmax/unembedding compute, and what happens to typical sequence lengths. Why did Llama 3 and Gemma accept these costs?
Embedding/unembedding params grow 4x to about 537M each at d=4096; per-step output head FLOPs quadruple; but text tokenizes into fewer tokens (~15% on English, more on code/multilingual), so total compute often drops.
Imagine the model has two big phone books at its ends, one for reading words in and one for choosing words out. Going from a 32k-word vocabulary to 128k-word makes each book four times bigger. That sounds expensive: more parameters to store, more work at the output to choose among more words. But the deeper part of the model, the dozens of attention and FFN layers, charge by the token. A bigger vocabulary means each chunk of text becomes fewer tokens, which means fewer trips through every layer. Llama 3 found that the extra cost at the ends was small compared to the savings spread across all the layers, especially for code and non-English text where the old 32k vocabulary wasted tokens splitting common words into pieces.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Vocabulary size is one of those architectural choices that looks like a fixed property of a model but is actually a deeply consequential tradeoff. Llama 2 shipped a 32k vocab in 2023. Llama 3 shipped a 128k vocab in 2024. Gemma 2 went further, to 256k. The trend is unambiguous, and the reasons are worth understanding because the same tradeoff shows up in every team's tokenizer decision.
This walkthrough lays out the three ledgers that change when you grow vocabulary, the arithmetic that explains why the trade usually pays off, and the language- and content-specific reasons frontier labs converged on large vocabs.
Mental model: vocab cost is at the two ends of the model. Sequence-length savings spread across the entire L-layer stack. The L-block bulk almost always beats the two-end ends, until you push V so high the embedding ends dominate the param budget.
Ledger 1: parameters
Embedding table
Shape [V, d_model]. At d = 4096:
- V=32k: 131M params
- V=128k: 524M params
- Delta: +393M
Unembedding head (untied)
Same shape, same growth. If untied, the model adds another 524M on top.
Tied vs untied
At 8B parameters, Llama 3 8B unties the head. The 524M extra params buy a specialised output projection that is worth more than the parameter cost at this scale. Smaller variants (Llama 3.2 1B, 3B) tie because the relative cost is too high.
Param fraction
On an 8B model with untied embeddings at d=4096:
- V=32k: embedding ends are ~3% of params
- V=128k: embedding ends are ~13% of params
On a 70B model with untied embeddings at d=8192 (note: d scales with model size), the embedding fraction stays at single digits. The cost scales with V but its impact scales as V / (L d^2), which shrinks for big models.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3 8B and 70B moved from Llama 2's 32k vocab to 128k, citing the compute and quality gains from compression.
- Gemma 2 ships a 256k vocab, even larger than Llama 3, optimised for multilingual coverage at the cost of bigger embedding ends.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the compression number split between English, code, and CJK?
English: ~15% fewer tokens. Code: 20-40% depending on language. CJK: often 50%+ because the smaller vocab forced byte-level fragmentation. The composition of your training mix determines the realised average.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Counting only the parameter cost and ignoring the compression effect. The per-token compute lives in all 32 layers; the vocab cost lives only at the two ends, so a small relative compression usually pays for itself.
60 second bullets to scan on the way to the call.
How embedding and unembedding params scale with vocab
Per-decoding-step output-head FLOPs versus per-token transformer block FLOPs
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.