Shorter sequences versus fatter model edges. A bigger vocabulary compresses text into fewer tokens, cutting attention cost (O(n squared)) and KV cache size. But the embedding matrix and lm_head both scale linearly with vocab_size, so doubling the vocabulary roughly doubles those parameter counts, slows the softmax, and creates rare tokens that may be under-trained. At 32K (Mistral, Llama 2) sequences are long but embeddings are compact. At 128K (Llama 3) sequences shrink and multilingual fertility improves at 4x embedding cost. At 256K (Gemma 2) you get the best compression but more tail risk. The 2026 sweet spot is 100K to 200K.
The core tradeoff your team should frame is **shorter sequences versus fatter model edges**. Each of the three vocabulary sizes illustrates a different point on that curve.
At 32K (Mistral 7B, Llama 2), the embedding matrix and lm_head stay small, but sequences are long. A paragraph that takes 400 tokens at 32K might compress to only 250 at 128K. Attention is O(n squared) in sequence length, so longer sequences cost significantly more per layer, and the KV cache is proportionally larger.
At 128K (Llama 3, Llama 4), sequences shrink meaningfully and multilingual fertility improves because there is room for non-Latin merges. The cost is a 4x larger embedding matrix and lm_head compared to 32K. On a 7B model that adds roughly 786M parameters.
At 256K (Gemma 2), you get the best compression and the shortest sequences, but rare tokens in the vocabulary may appear so infrequently in training that their embeddings are under-trained. The softmax over 256K logits is also measurably slower per token.
The 2026 consensus is that 100K to 200K is the sweet spot for general LLMs trained on multi-trillion token corpora. Below that, the attention savings go unrealized; above that, rare token problems grow faster than compression gains.
**Vocabulary size** trades sequence length against model parameter cost, and your team should frame the debate around exactly that axis.
The upside of a bigger vocabulary is shorter sequences. Each token covers more text, so the same content compresses into fewer positions. Attention is O(n squared) per layer, so halving the sequence cuts attention compute by 4x. KV cache memory scales linearly with sequence length, so shorter sequences also save inference memory. Both numbers matter for long context applications.
The downside has three parts. First, the embedding matrix is shaped vocab_size by d_model, and the lm_head is shaped d_model by vocab_size. Doubling vocabulary doubles both of these. On a 7B model with d_model of 4096, going from 32K to 128K adds roughly 786M parameters. Second, the softmax is computed over the full vocabulary every token, adding per-token latency proportional to vocab_size. Third, rare tokens in larger vocabularies may have under-trained embeddings that produce surprising outputs.
2026 production sweet spots:
- **tiktoken cl100k_base**: 100K vocab, GPT-4 era.
- **tiktoken o200k_base**: 200K vocab, GPT-4o through GPT-5.5.
- **Llama 3 and 4**: 128K vocab.
- **Mistral 7B family**: 32K vocab.
- **Gemma 2**: 256K vocab (notably large for multilingual coverage).
The trend across 2023 to 2026 has been clearly upward, driven by longer context windows making attention savings more valuable, multilingual needs requiring vocabulary headroom, and larger training corpora supporting the bigger vocabularies.
Quick reference
TL;DR
Bigger vocab compresses text into shorter sequences (cheaper attention) but fattens the embedding matrix, lm_head, and softmax. The 2026 sweet spot for general LLMs is 100K to 200K.
Picture a dictionary you carry around to read every book. A tiny dictionary fits in your pocket but you have to look up every other word, so reading is slow. A huge dictionary lets you read fast because almost every word is one lookup, but the book itself is now twice as heavy in your backpack. Vocabulary size in a language model is the same choice. A small vocabulary breaks text into many small tokens, making the model work through more positions, and the attention math gets expensive. A big vocabulary compresses text into fewer tokens, so the model processes it in fewer steps. But the model also carries one slot per vocabulary entry in two large matrices, so a huge vocabulary makes the model bigger and the final prediction step slower.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Training a small specialist model on a narrow English corpus. A 32K to 65K vocabulary is usually sufficient; 200K wastes parameters on tokens you will never see and creates under-trained embeddings.
Targeting tight inference budgets where embedding matrix memory is constrained. On edge devices or tight VRAM budgets, the embedding matrix is a meaningful fraction of total memory; a smaller vocabulary leaves more room for transformer weights.
Pretraining on a small dataset of single-digit billions of tokens. Larger vocabularies need more data to train the rare tokens well; on a small dataset the tail is permanently under-trained.
Quick scan · 2–4 min
Everything important, quickly.
4 min: state the core tradeoff (compression versus parameter cost), walk the three downside components (embedding, lm_head/softmax, under-training), anchor with 2026 production numbers and model examples, and explain why the trend has moved toward larger vocabularies.
Real-world examples
Real products, models, and research that use this idea.
1tiktoken cl100k_base ships a 100K vocabulary for GPT-4 era models; o200k_base ships 200K for GPT-4o, o1, o3, GPT-5, and GPT-5.5.
2Llama 3 and Llama 4 use a 128K BPE vocabulary, quadrupled from Llama 2's 32K SentencePiece.
3Gemma 2 uses a 256K SentencePiece Unigram vocabulary, deliberately large for multilingual headroom.
4Mistral 7B and Mistral Large early variants use a 32K SentencePiece BPE vocabulary; later Mistral Large models moved to a larger vocab to match the industry trend.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QDoubling the vocabulary roughly halves sequence length. Why does that not perfectly cancel out the doubled embedding cost?
Sign in or go Pro to see the approach for this follow-up.
QHow do production teams decide if their vocabulary is the right size?
Sign in or go Pro to see the approach for this follow-up.
QWhy does Gemma use 256K while Mistral models use 32K?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Thinking bigger vocabulary is strictly better. The embedding matrix and lm_head both scale linearly with vocab_size, doubling the vocabulary roughly doubles those parameters and adds real cost to every forward pass.