Take the same semantic content (say a 100 word English passage) and translate it to Spanish, Hindi, and Japanese. Tokenize each translation with o200k_base. Predict the approximate ratio of token counts for each language relative to English (English baseline = 1.0). Then briefly explain why these ratios improved on o200k_base versus cl100k_base.
On o200k_base: English 1.0, Spanish ~1.2, Hindi 2-3, Japanese 1.5-2. Better than cl100k_base because o200k_base added dedicated CJK and Devanagari tokens. Validate with tiktoken on real samples.
Imagine four people from different countries trying to send the same message using a shared shorthand notebook. The notebook was designed mostly with English in mind. The English speaker writes the message in a few quick squiggles. The Spanish speaker is also pretty fast because their alphabet is similar. The Hindi and Japanese speakers find that their writing systems are not really in the notebook, so they have to spell things out in tiny pieces, which takes much more space. A newer notebook (o200k_base) added shortcuts for Hindi, Japanese, and Chinese characters, so they now spell things out in fewer pieces than before, but they are still slower than English. The numbers: English baseline of 1, Spanish about 1.2, Hindi about 2.5, Japanese about 1.7. These ratios determine how much you pay per message in each language.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Tokenizer fertility per language is one of the higher-leverage facts an LLM engineer can internalize, because it directly determines per-user cost and per-user context budget in any multilingual deployment. The headline numbers are easy: on o200k_base, English is 1.0, Spanish 1.1-1.3, Japanese 1.5-2, Hindi 2-3. The underlying story is about how OpenAI allocated 200K vocabulary slots, why some languages benefited more than others, and what production teams should actually do with these numbers.
The rest of this explanation walks the writing-system explanation for why ratios vary, what changed between cl100k_base and o200k_base, why the improvement was uneven across languages, the operational implications for cost models and context budgets, and the production discipline for validating ratios on real samples.
Writing system as the primary driver of fertility
Tokenizer fertility for a language is determined by two things: how the BPE merge table allocated tokens to patterns common in that language, and how the language's writing system maps to UTF-8 bytes. Both factors compound.
Latin-script languages (English, Spanish, French, Italian, German, Portuguese) share the ASCII range plus a small set of accented characters. The byte representation is 1-2 bytes per character. BPE merge tables trained on English-dominant corpora learned merges that cover Latin patterns broadly, so Spanish or French content reuses many of the same tokens as English with small per-language additions. Result: fertility close to English, typically 1.1-1.3x.
German sits at the upper end of Latin (1.3-1.5x) because compound words like 'Donaudampfschifffahrtsgesellschaftskapitän' do not have dedicated tokens and fragment into pieces.
CJK (Chinese, Japanese, Korean) uses ideographs and syllabaries that are 3-4 bytes per character in UTF-8. Without dedicated merges, each kanji/hangul character would tokenize as 3-4 byte tokens, giving fertility 5-10x English. Modern tokenizers like o200k_base added dedicated CJK tokens for common characters, dropping fertility to 1.5-2x. Japanese benefits from kanji sharing with Chinese; Korean (hangul) has less cross-language sharing.
Devanagari (Hindi, Marathi, Nepali) and other Indic scripts (Tamil, Telugu, Bengali) are 3 bytes per character in UTF-8. Plus, Indic scripts have complex grapheme clusters where multiple codepoints combine into a single visual character. Without dedicated merges, fertility runs 4-6x English. o200k_base added dedicated Devanagari tokens for common Hindi syllables, dropping Hindi to 2-3x. Lower-resource Indic languages got less attention and remain at 2.5-3.5x.
Arabic and Hebrew are 2 bytes per character in UTF-8 but have complex morphology (root-based word formation) that fragments under BPE without language-specific design. o200k_base improved Arabic to ~2.0x; Hebrew is similar.
Vietnamese uses a Latin-derived alphabet with extensive diacritics; the diacritics fragment under naive BPE, pushing fertility to 2-3x despite being Latin-script.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- tiktoken.get_encoding('o200k_base') on Hindi medical text in 2026 returns roughly 2.5 tokens per word versus 1.3 for English on the same content domain.
- OpenAI's GPT-4o tokenizer announcement explicitly highlighted improved multilingual fertility as a design goal, citing CJK and Devanagari as primary targets.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour service expands to Tamil and Bengali users. What do you expect to happen to per-ticket cost?
Tamil and Bengali are in the 2.5-3.5x English range on o200k_base because they are lower-resource languages that received less dedicated token budget. Expect per-ticket input cost to roughly triple for these tenants versus an English-monolingual baseline. Mitigations: route to open-weight models with stronger Indic coverage (Llama 3 multilingual variants, Qwen, or specialized Indic models like Sarvam), or extend o200k_base via fine-tuning if you control the model.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a single 'tokens per word' multiplier applies across all languages, when fertility varies 1.0 to 3.0 within o200k_base alone.
60 second bullets to scan on the way to the call.
Approximate o200k_base fertility ratios for English, Spanish, Hindi, Japanese.
Why Latin-script languages tokenize close to English.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.