You are building a 7B model for 25 languages including Hindi, Arabic, and Swahili. Design a fair tokenizer and name what it costs.

Short answer·Hard·4.0 · 0·~3 min·Relevant atAi4bharat Google Sarvam

Attempt it

Free · 2 AI evals / day

Five design axes, each with a fairness/cost tradeoff. **Vocab size.** 200K-256K. Smaller forces fragmentation on Hindi, Swahili, Vietnamese (3-5x English fertility). Larger inflates embedding+lm_head parameters proportional to d_model. For a 7B model with d_model=4096, 256K vocab adds about 2.1B parameters (embedding+lm_head combined), about 30 percent of the model. **Training corpus mixture.** Sampling temperature 0.3-0.5. Naive proportional sampling makes English dominant and starves low-resource languages of merges. Temperature 0.3 upsamples them; English fertility worsens slightly but mean fairness improves. **Algorithm.** Byte-level BPE: guaranteed Unicode coverage via byte fallback, fast inference, widely supported. Unigram-LM is the robustness-focused alternative; consider if compute allows. **Normalization.** NFC, not NFKC. NFC preserves script-specific characters that matter for Arabic presentation forms and Mandarin full-width vs half-width. NFKC over-normalizes. **Evaluation.** Per-language fertility (tokens per word) AND single-token retention rate (STRR) AND downstream accuracy per language. Report each language separately, never just the mean. Token Tax research showed fertility predicts downstream accuracy. **The tradeoff to surface.** A fair tokenizer costs parameters and softmax latency. Be explicit about the budget. Production teams routinely under-budget for the parameter cost and find at deploy that the multilingual model is meaningfully larger than expected.

**Five design axes.** **Vocab size: 200K-256K.** Smaller forces fragmentation on low-resource languages (Hindi/Swahili at 3-5x English). Larger has diminishing returns. For 7B at d_model=4096, 256K vocab adds ~2.1B params in embedding+lm_head, ~30 percent of model. Budget explicitly. **Corpus mixture: sampling temperature 0.3-0.5.** Web prevalence puts English at 50-70 percent; naive sampling produces an English-dominant BPE that starves low-resource languages of merges. Temperature 0.3 upsamples them. Tradeoff: English worsens 5-10 percent; low-resource languages improve 30-50 percent. **Algorithm: byte-level BPE.** Production default. Byte fallback for coverage, fast inference, broad ecosystem. Unigram-LM with subword regularization is the robustness-focused alternative; consider for morphologically rich languages if compute allows. **Normalization: NFC, not NFKC.** NFC preserves script-specific characters that matter semantically (Arabic presentation forms, Mandarin full-width vs half-width). NFKC over-normalizes. **Evaluation: per-language metrics.** (1) Fertility (tokens/word). (2) STRR (single-token retention rate). (3) Downstream task accuracy per language. Never report only the mean; fairness lives in the variance. Token Tax research (Petrov 2023, Ahia 2023) showed fertility predicts downstream accuracy. **The tradeoff.** Fair tokenizer costs ~30 percent of model parameters in vocab inflation and softmax latency grows linearly per step. The English-monolingual baseline at 50K vocab adds only 200M params; you are spending 2B extra on multilingual fairness. Worth it for a genuine multilingual product; overkill for English-dominant deployment.

Quick reference

TL;DR

200K-256K byte-level BPE, sampling temperature 0.3-0.5, NFC, per-language fertility+STRR+downstream eval. Fairness costs ~30 percent of model parameters and softmax latency grows linearly.

Memory aid

Easy to grasp

Imagine designing a phonebook for 25 different countries to share. If you give 1,000 entries to English alone, every other language has to spell things out letter by letter. If you give a fair share to each language, the phonebook becomes huge (which costs paper and time to flip through), but everyone gets quick lookups. A fair multilingual tokenizer is the second design. You spend more parameters on the bigger vocabulary, and the model takes slightly longer at every step because it has more options to choose from at the end. In return, speakers of Hindi, Swahili, Vietnamese, and other under-served languages get the same quality of service English speakers get. The cost of fairness is real (~30 percent of model parameters); the benefit is real too. Production teams decide whether to pay it based on who their users are.

Key concepts

Multilingual fairness Sampling temperature Byte-level BPE STRR

Detailed answer & concept explanation

~9 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Open with the framing that a fair multilingual tokenizer is a parameter and latency budget decision disguised as a tokenization decision. Walk five design axes: vocab 200K-256K, sampling temperature 0.3-0.5, byte-level BPE, NFC normalization, per-language fertility+STRR+downstream eval. Explicitly surface the cost: 256K vocab on a 7B at d_model=4096 adds ~2.1B params (30 percent of model) and softmax latency grows linearly. Close with the production examples (Llama 3 128K, Qwen 3.5 152K, Gemma 2 256K) and the Token Tax research that motivates per-language evaluation.

Real-world examples

Real products, models, and research that use this idea.

Llama 3 and Llama 3.1 use a 128K byte-level BPE vocabulary, a middle ground that improves over Llama 2's 32K but does not fully achieve fairness for low-resource languages.
Qwen 3.5 ships a 152K BPE optimized for strong Mandarin coverage and reasonable English performance.
Gemma 2 uses a 256K SentencePiece-style BPE, the largest in the open-weight class as of 2026, with explicit multilingual fairness design.
The 'Token Tax' research (Petrov 2023, Ahia 2023) demonstrated empirically that fertility predicts downstream task accuracy across languages, motivating the per-language fertility metric.
Sarvam AI, OpenHathi, and BharatGPT are 2025-2026 production examples of Indic-specialized tokenizers that extend or replace English-centric vocabularies for stronger Hindi/Tamil/Bengali coverage.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QSampling temperature 0.3 upsamples Swahili by 50x its web prevalence. Doesn't that produce a tokenizer that does badly on English where most of the actual product traffic is?

QWhy does fertility correlate with downstream accuracy across languages?

QHow does the softmax cost per step argument change for reasoning models that emit many output tokens?

QHow do you defend a 'fair' tokenizer choice to a product team that wants a smaller, cheaper model?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Optimizing for English fertility and reporting only the mean, hiding that low-resource languages fragment at 3-5x English rates and pay the 'token tax' in cost, context, and downstream quality.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.