What fraction of a 7B transformer lives in its embedding table at vocab 128k?

MCQ·Medium·4.0 · 0·~1 min·Relevant atMistral AI NVIDIA

Attempt it

**Around 7-8% of a 7B model lives in the embedding table when vocab is 128k.** Do the math directly: - Embedding table shape: `vocab_size x d_model = 128000 x 4096` - Parameter count: `128000 * 4096 = 524,288,000` (~524M) - Fraction of 7B: `524M / 7000M = 7.5%` That's just the **input embedding**. If the output head (the LM head that projects the final hidden state to vocab logits) is NOT tied to the input embedding, it costs the same again: another 524M parameters, bringing the embedding pair to ~1.05B, or 15% of a 7B model. **Why the answer isn't 'about 1%'.** A 1% of model embedding would require vocab ~17k at `d_model = 4096`, which is smaller than any modern tokenizer. **Why the answer isn't 'about 25% or 50%'.** The transformer stack itself (32 layers of attention + FFN at d_model 4096) is roughly 6 billion parameters in a 7B model. The embedding can't dominate that unless vocab climbs into the millions. This is why modern LLMs care about tokenizer choice: a larger vocab gives better compression and fewer tokens per word, but each thousand tokens of vocab adds 4M parameters to the embedding.

**About 7.5%, call it 7-8%. The math is concrete and worth memorizing.** Llama 3.1 8B uses vocab 128256 and `d_model = 4096`. The input embedding matrix is `128256 x 4096 ≈ 525M parameters`. On an 8B model that's roughly 6.6% of the total. If the output head is untied (Llama 3.1 keeps them untied), another 525M lives in the LM head, bringing the embedding pair to ~1.05B or 13% of the model. **Why this matters in practice.** - **Tokenizer choice has architectural weight.** A 32k vocab (Llama 2 era) cost only 131M params for the input embedding: about a quarter as much. The choice to grow vocab from 32k to 128k added ~400M params to the embedding pair on its own. - **Embedding tying flips with scale.** At small scale (sub-1B), tying input and output embeddings is the default to save parameters. At frontier scale, untying gives a measurable quality bump and the extra ~7% parameter cost is negligible compared to the quality gain. - **Output head dominates inference latency for some workloads.** At each generation step, the output head matmul (`hidden_dim x vocab`) is one of the largest single ops. At vocab 128k that's a 4096x128000 matmul per step. **The MCQ distractors are calibrated.** A: 1% would only apply if vocab were ~16k. C: 25% would require vocab ~430k, larger than any production tokenizer. D: 50% would require vocab approaching 1M, which only experimental ByT5 style byte level models approach.

Quick reference

TL;DR

128k vocab times d_model 4096 is ~524M parameters in the input embedding alone, about 7.5% of a 7B model. Untying the output head doubles the cost to ~15%.

Memory aid

Easy to grasp

Imagine a giant phone book where every word the model knows has its own page, and each page is filled with a long list of numbers that describe what the word means. The phone book is huge: 128,000 pages times 4,096 numbers per page comes out to about 524 million numbers. On a 7 billion number model, that phone book alone is 7.5% of the whole thing. Not a rounding error, but not a black hole either: most of the model is still the stack of thinking layers above it. If you also keep a second phone book at the back of the model for turning answers into word scores, the pair takes about 15% of the total. Sharing one phone book for both ends (called tying) saves that 7.5% at a tiny cost in quality.

Key concepts

Transformer architecture Tokenization Embeddings

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: do the multiplication, contrast tied vs untied embeddings, explain why Llama 3 moved to 128k vocab, and discuss how the fraction scales with model size.

Side-by-side

Model	Vocab	d_model	Embedding params	% of total
Mistral 7B (tied)	32k	4096	131M	1.9%
Llama 2 7B (tied)	32k	4096	131M	1.9%
Llama 3.1 8B (untied)	128k	4096	1.05B	13%
Llama 3.1 70B (untied)	128k	8192	2.1B	3%
Gemma 4 (untied)	256k	2048	1.05B	varies

Real-world examples

Real products, models, and research that use this idea.

Llama 3.1 8B: vocab 128256, d_model 4096, embedding pair ~1.05B (13% of 8B).
Llama 3.1 70B: vocab 128256, d_model 8192, embedding pair ~2.1B (3% of 70B): fraction shrinks at scale.
Mistral 7B: vocab 32000, d_model 4096, embedding pair ~131M (tied, so 1.9% of 7B).
Gemma 4: vocab 256000, d_model 2048, embedding pair ~1.05B; vocab grew while d_model shrank.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy doesn't the embedding fraction scale linearly with model size?

QHow would you choose vocab size for a new LLM?

QWhat does the embedding table actually learn?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating embeddings as negligible because they are a lookup. They're not: at 128k vocab they're 7-8% of a 7B model, and at 256k plus vocab the cost climbs further.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.