What fraction of a 7B transformer lives in its embedding table at vocab 128k?

MCQ·Medium·4.0 · 0·~1 min·Asked atCrewai NVIDIA Workday·Relevant atMistral AI

Attempt it

**Around 7-8% of a 7B model lives in the embedding table when vocab is 128k.** Do the math directly: - Embedding table shape: `vocab_size x d_model = 128000 x 4096` - Parameter count: `128000 * 4096 = 524,288,000` (~524M) - Fraction of 7B: `524M / 7000M = 7.5%` That's just the **input embedding**. If the output head (the LM head that projects the final hidden state to vocab logits) is NOT tied to the input embedding, it costs the same again: another 524M parameters, bringing the embedding pair to ~1.05B, or 15% of a 7B model. **Why the answer isn't 'about 1%'.** A 1% of model embedding would require vocab ~17k at `d_model = 4096`, which is smaller than any modern tokenizer. **Why the answer isn't 'about 25% or 50%'.** The transformer stack itself (32 layers of attention + FFN at d_model 4096) is roughly 6 billion parameters in a 7B model. The embedding can't dominate that unless vocab climbs into the millions. This is why modern LLMs care about tokenizer choice: a larger vocab gives better compression and fewer tokens per word, but each thousand tokens of vocab adds 4M parameters to the embedding.

**About 7.5%, call it 7-8%. The math is concrete and worth memorizing.** Llama 3.1 8B uses vocab 128256 and `d_model = 4096`. The input embedding matrix is `128256 x 4096 ≈ 525M parameters`. On an 8B model that's roughly 6.6% of the total. If the output head is untied (Llama 3.1 keeps them untied), another 525M lives in the LM head, bringing the embedding pair to ~1.05B or 13% of the model. **Why this matters in practice.** - **Tokenizer choice has architectural weight.** A 32k vocab (Llama 2 era) cost only 131M params for the input embedding: about a quarter as much. The choice to grow vocab from 32k to 128k added ~400M params to the embedding pair on its own. - **Embedding tying flips with scale.** At small scale (sub-1B), tying input and output embeddings is the default to save parameters. At frontier scale, untying gives a measurable quality bump and the extra ~7% parameter cost is negligible compared to the quality gain. - **Output head dominates inference latency for some workloads.** At each generation step, the output head matmul (`hidden_dim x vocab`) is one of the largest single ops. At vocab 128k that's a 4096x128000 matmul per step. **The MCQ distractors are calibrated.** A: 1% would only apply if vocab were ~16k. C: 25% would require vocab ~430k, larger than any production tokenizer. D: 50% would require vocab approaching 1M, which only experimental ByT5 style byte level models approach.

TL;DR

128k vocab times d_model 4096 is ~524M parameters in the input embedding alone, about 7.5% of a 7B model. Untying the output head doubles the cost to ~15%.

Memory aid

Easy to grasp

Imagine a giant phone book where every word the model knows has its own page, and each page is filled with a long list of numbers that describe what the word means. The phone book is huge: 128,000 pages times 4,096 numbers per page comes out to about 524 million numbers. On a 7 billion number model, that phone book alone is 7.5% of the whole thing. Not a rounding error, but not a black hole either: most of the model is still the stack of thinking layers above it. If you also keep a second phone book at the back of the model for turning answers into word scores, the pair takes about 15% of the total. Sharing one phone book for both ends (called tying) saves that 7.5% at a tiny cost in quality.

Key concepts

Transformer architecture Tokenization Embeddings

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Embedding tables look like an implementation detail and behave like a major architectural parameter sink. At vocab 128k and d_model = 4096, the input embedding alone is half a billion parameters: about 7-8% of an 8B model. Once you add the untied output head, the embedding pair can climb to 13% of the model. This is the kind of number that surprises engineers who think of embeddings as 'just a lookup table' but takes only one multiplication to verify.

This question tests whether you can do back of envelope parameter math fluently. The senior answer goes further: it connects the embedding cost to tokenizer choice, embedding tying decisions, and serving time latency considerations.

The arithmetic and the four MCQ distractors

The computation is one multiplication. Embedding shape is vocab_size x d_model. At vocab 128256 (Llama 3.1's tokenizer) and d_model = 4096, that's:

128256 \times 4096 \approx 525 \text{ million parameters}

For an 8B model, that's 525M / 8030M = 6.5%. Call it 7-8% with the output head accounting for varying tied/untied choices. That's the answer.

Why option A (1%) is wrong. 1% of an 8B model is 80M parameters. To get the input embedding down to 80M at d_model = 4096, you'd need vocab ~19k: smaller than any modern production tokenizer. Option A reflects a 2018 era intuition where vocab 30k was standard and d_model was 768; at those numbers the embedding was a few percent of a 110M BERT.

Why option C (25%) is wrong. 25% of 8B is 2B. To reach 2B parameters in the input embedding alone you'd need vocab ~490k, larger than any production tokenizer. Gemma 4's 256k vocab is the upper end of current practice and still only produces a 525M embedding at d_model = 2048.

Why option D (50%) is wrong. 50% would require either vocab approaching 1 million OR a model much smaller than 7B. Neither matches the question's premise.

The correct answer, 7-8%, sits comfortably in the middle: significant enough that vocab choice is an architectural decision but small enough that the transformer stack still dominates.

Why Llama 3 grew vocab from 32k to 128k

Tying vs untying input and output embeddings

Where the embedding shows up at serving time

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Model	Vocab	d_model	Embedding params	% of total
Mistral 7B (tied)	32k	4096	131M	1.9%
Llama 2 7B (tied)	32k	4096	131M	1.9%
Llama 3.1 8B (untied)	128k	4096	1.05B	13%
Llama 3.1 70B (untied)	128k	8192	2.1B	3%
Gemma 4 (untied)	256k	2048	1.05B	varies

Real products, models, and research that use this idea.

Llama 3.1 8B: vocab 128256, d_model 4096, embedding pair ~1.05B (13% of 8B).
Llama 3.1 70B: vocab 128256, d_model 8192, embedding pair ~2.1B (3% of 70B): fraction shrinks at scale.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy doesn't the embedding fraction scale linearly with model size?

Embedding cost is vocab * d_model, which scales with d_model (linearly with width) but not with depth (number of layers). The transformer stack scales with depth * d_model^2 (quadratically with width). At larger models, the stack grows faster than the embedding.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating embeddings as negligible because they are a lookup. They're not: at 128k vocab they're 7-8% of a 7B model, and at 256k plus vocab the cost climbs further.

60 second bullets to scan on the way to the call.

The vocab x d_model formula for embedding parameter count
What fraction of a 7B / 8B / 70B model the embedding takes at vocab 128k

Primary sources. Browse if you want the original framing.