Predict the embedding table parameter count for a 32k vocab at d_model 512

Predict output·Easy·4.0 · 0·~2 min·Asked atBanana Dev Cerebras Shopify

Attempt it

A teammate is sizing a small decoder-only model: vocab_size = 32,000 and d_model = 512, with the output head tied to the input embedding. Predict how many trainable parameters the embedding table contributes.

**The embedding table** is a single matrix that maps each token id to a `d_model`-dimensional vector. Its shape is `[vocab_size, d_model]`, so its parameter count is exactly the product of those two numbers. For the question's numbers: 32,000 rows (one per vocabulary entry) times 512 columns (the model's residual stream width) gives 16,384,000 parameters, or about 16.4 million. **Weight tying** is the standard practice of using the same matrix for both the input embedding and the output (unembedding) head. The input lookup picks one row per token; the output projection multiplies the residual stream by the transpose of the same matrix to produce logits over the vocabulary. Because the same parameters serve both roles, the head adds zero extra parameters. **Why this matters.** For a small model, the embedding table can be a noticeable fraction of total parameters. A 100M-parameter model with vocab 32k and `d_model` 512 spends 16.4M of those, or roughly 16%, just on embeddings. Weight tying keeps this from doubling. For larger models with `d_model` of several thousand, the embedding fraction shrinks but is never zero.

**Compute directly.** The embedding table is one matrix of shape `[vocab_size, d_model]`. Parameter count: `32000 * 512 = 16,384,000`, about 16.4M. **Weight tying** ties the output head's weights to the embedding table. Without tying, the unembedding would be a separate matrix of shape `[d_model, vocab_size]` with the same parameter count. Tying halves the embedding-related parameter budget by reusing one matrix. $$|\theta_\text{embed}| = V \cdot d_\text{model}$$ **Where this fits in a small decoder.** A 100M-parameter scratch model with this configuration spends roughly 16% of its budget on embeddings. The remaining 84% goes to attention QKV projections and output projections, FFN layers, and norms. As `d_model` grows, the embedding fraction shrinks because the FFN grows quadratically with `d_model` (4 * d_model squared per layer is the dominant cost) while embeddings grow only linearly. **Why the original transformer tied embeddings.** Press and Wolf (2017) showed that tying improves both efficiency and quality for small models. The tied matrix sees both input gradient (via the lookup) and output gradient (via the logit head), which actually regularizes the learned representations. Modern models (GPT-2, Llama, Mistral) commonly tie embeddings except at very large scales where the head is sometimes decoupled. **Quick sanity check.** Always verify by multiplying vocab times d_model and asking 'is this a sensible fraction of my model's total parameters?' If it is more than 30%, your model is too small for its vocabulary or your vocabulary is too large for the model size.

TL;DR

The embedding table is a matrix of shape vocab times d_model, here 32000 times 512 equals 16,384,000 parameters. Tying the output head reuses the same matrix at zero extra cost.

Memory aid

Easy to grasp

Imagine a giant dictionary with 32,000 entries, where each entry stores a list of 512 numbers describing that word. The total number of numbers in the dictionary is just the number of rows times the numbers per row, like counting cells in a spreadsheet with 32,000 rows and 512 columns. The model also needs to translate its internal thoughts back into one of those 32,000 words at the end. Instead of building a second dictionary, it reuses the same one in reverse, which is called weight tying. Two jobs, one table, zero extra parameters.

Key concepts

Transformer Architecture Tokenization

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The embedding table is the entry point of a transformer. Every input token is a lookup into a fixed matrix of shape [vocab_size, d_model], returning one row per token. The output side mirrors this: the final residual stream is multiplied by the transpose of the same matrix (under weight tying) to produce logits over the vocabulary. The parameter count is exactly the product of the two dimensions: 32,000 by 512 in this question, or 16,384,000 parameters.

This is a basic computation, but it surfaces three deeper questions worth understanding: how the embedding fits into the overall parameter budget, why weight tying matters, and when modern large models choose to untie the head despite the cost. The deep dive walks through each.

The takeaway will be that embedding sizing is one of the earliest and most consequential design choices in any decoder architecture. Get it wrong and your model spends most of its budget on a lookup table; get it right and the table becomes a small, fixed cost that scales gracefully as the model grows.

The arithmetic and what it means

The shape. An embedding table is a single 2D parameter matrix E of shape [V, d] where V is the vocabulary size and d is d_model. Each row is the learned representation of one token id. The forward pass for a token at position t is just embedding[t] = E[token_id_t], a row lookup; no matrix multiplication, no activation.

The count. Parameters in E equal V * d. For the question: 32,000 times 512 equals 16,384,000. About 16.4M parameters live in the embedding table.

In context. A small decoder with this vocabulary and d_model might have around 100M parameters total, depending on layer count and FFN expansion. The embedding alone is about 16% of that budget. If you doubled vocabulary to 64k for better tokenization, the embedding would consume 32% of the budget, which is too much. If you instead doubled d_model to 1024, the embedding grows to 32M but the rest of the model grows much faster (FFN goes as d_model squared) so the embedding fraction shrinks.

The right scaling. Vocabulary should be proportional to model size. Small models (under 1B params) use 32k vocabularies. Mid-size models (7B-30B) use 32k-128k. Large models (70B+) use 128k or larger. The cost of a bigger vocabulary in embedding parameters is real but justified at scale because the rest of the model can afford it.

Weight tying: same matrix, two jobs

Embedding fraction across the model size spectrum

Tying versus untying at scale

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

GPT-2 (vocab 50257, d_model 768) ties embeddings and spends roughly 38.6M of its 124M parameters on the table, about 31% of the budget
Llama 3 8B (vocab 128k, d_model 4096) does NOT tie embeddings; the embedding plus head together cost roughly 1.05B parameters, about 13% of the 8B total

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the embedding fraction change if you scale d_model from 512 to 4096 while keeping vocab at 32k?

Embedding scales linearly with d_model, but FFN scales with d_model squared and attention with d_model squared. The embedding fraction shrinks from order 16% at small d_model to single digits or less at d_model in the thousands.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Adding a separate parameter count for the unembedding head, forgetting that weight tying makes input and output share the same matrix.

60 second bullets to scan on the way to the call.

Why the embedding table shape is exactly vocab times d_model
What weight tying does to the head parameter count

Primary sources. Browse if you want the original framing.