You are building a 7B model for 25 languages including Hindi, Arabic, and Swahili. Design a fair tokenizer and name what it costs.
200K-256K byte-level BPE, sampling temperature 0.3-0.5, NFC, per-language fertility+STRR+downstream eval. Fairness costs ~30 percent of model parameters and softmax latency grows linearly.
Imagine designing a phonebook for 25 different countries to share. If you give 1,000 entries to English alone, every other language has to spell things out letter by letter. If you give a fair share to each language, the phonebook becomes huge (which costs paper and time to flip through), but everyone gets quick lookups. A fair multilingual tokenizer is the second design. You spend more parameters on the bigger vocabulary, and the model takes slightly longer at every step because it has more options to choose from at the end. In return, speakers of Hindi, Swahili, Vietnamese, and other under-served languages get the same quality of service English speakers get. The cost of fairness is real (~30 percent of model parameters); the benefit is real too. Production teams decide whether to pay it based on who their users are.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Designing a fair multilingual tokenizer for a 7B model serving 25 languages is one of the more architecturally consequential decisions a model team makes. The headline question (what algorithm, what vocab size) has well-known answers. The harder question, the one that separates a thoughtful design from a generic one, is what fairness costs and how to defend that cost to a product team that wants both fairness and a small model.
The correct framing is that fairness is a parameter and latency budget decision. A fair multilingual tokenizer adds ~2.1B parameters in embedding+lm_head for a 7B model at d_model=4096 with a 256K vocab. That is ~30 percent of the model. Softmax latency at the lm_head grows linearly with vocab size, so per output token inference cost also rises. These are not hidden costs; they are explicit tradeoffs that the team has to budget for.
The rest of this explanation walks each design axis in detail (vocab size, corpus sampling, algorithm, normalization, evaluation), names the production examples in 2026, and closes with the harder question of how to defend the cost to a non-technical audience.
Vocab size: 200K-256K and the parameter budget
Vocab size is the single most consequential decision. It determines fertility floor across languages, embedding parameter count, lm_head parameter count, and softmax latency per output token.
Lower bound: 200K. Below 200K, low-resource languages start to fragment unacceptably. Hindi at 100K vocab is 4-6x English fertility; at 200K it drops to 2-3x. Swahili and Vietnamese show similar patterns. The fragmentation hits a phase change around 200K because that is roughly when BPE training has enough budget to learn meaningful merges for less-frequent languages.
Upper bound: 256K. Above 256K, fertility improvement plateaus for most languages while parameter cost continues to grow linearly. The marginal benefit per additional 50K vocab tokens drops below the marginal cost. Some teams go higher (Gemma 2 ships 256K, some research models reach 400K), but 256K is the practical sweet spot for production 7B-class models in 2026.
Parameter cost. For a 7B model with d_model=4096, the embedding matrix is vocab_size * 4096 parameters and the lm_head is the same (often tied weights but separately budgeted). A 256K vocab is 256,000 * 4096 = ~1.05B params per matrix, or ~2.1B combined. For a nominally 7B model, this is ~30 percent of the parameter budget going to vocab.
A 50K English-monolingual vocab is 50K * 4096 = ~200M params per matrix, or ~400M combined. Going multilingual from there costs ~1.7B extra parameters for the same model otherwise. The 'fair multilingual 7B' is structurally a 9B model.
Softmax cost. At each output token, the model computes logits over the full vocabulary, which is an O(vocab_size * d_model) matrix multiply. Per-token latency for a 256K vocab is ~5x what it is for a 50K vocab on the same hardware. Output-heavy workloads (long generations, reasoning chains) feel this most.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3 and Llama 3.1 use a 128K byte-level BPE vocabulary, a middle ground that improves over Llama 2's 32K but does not fully achieve fairness for low-resource languages.
- Qwen 3.5 ships a 152K BPE optimized for strong Mandarin coverage and reasonable English performance.
What an interviewer would ask next. Try answering before peeking at the approach.
QSampling temperature 0.3 upsamples Swahili by 50x its web prevalence. Doesn't that produce a tokenizer that does badly on English where most of the actual product traffic is?
English fertility worsens by ~5-10 percent under temperature 0.3 sampling, not catastrophically. The product question is whether the worsening is acceptable in exchange for fair Swahili. For a primarily English deployment with occasional Swahili, temperature 0.5 or 0.7 is a more moderate choice. For a 25-language fairness-first deployment, temperature 0.3 is right. The choice is a product-strategic decision masquerading as a hyperparameter.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Optimizing for English fertility and reporting only the mean, hiding that low-resource languages fragment at 3-5x English rates and pay the 'token tax' in cost, context, and downstream quality.
60 second bullets to scan on the way to the call.
Why 200K-256K is the right vocab range for a 25-language 7B model.
How sampling temperature on training corpus changes which languages BPE learns merges for.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.