Take the same semantic content (say a 100-word English passage) and translate it to Spanish, Hindi, and Japanese. Tokenize each translation with o200k_base. Predict the approximate ratio of token counts for each language relative to English (English baseline = 1.0). Then briefly explain why these ratios improved on o200k_base versus cl100k_base.
The ratios on o200k_base, the tokenizer for GPT-4o, GPT-5, and GPT-5.5:
- English: 1.0 (baseline)
- Spanish: ~1.1-1.3
- Hindi: ~2-3
- Japanese: ~1.5-2
The pattern follows the writing system. Spanish uses the Latin alphabet, so it shares most token coverage with English; a few extra tokens for accented characters and Spanish-specific patterns. Hindi uses Devanagari, an entirely different script with no Latin overlap; on older tokenizers Devanagari fell back to byte-level sequences. Japanese mixes kanji, hiragana, and katakana, three writing systems in one language.
**Why the ratios improved on o200k_base versus cl100k_base.** OpenAI doubled the vocabulary from 100K to 200K when designing the GPT-4o tokenizer, and they explicitly allocated the extra 100K tokens to multilingual coverage. CJK (Chinese, Japanese, Korean) got dedicated multi-character tokens. Devanagari got dedicated tokens for common Hindi patterns. Arabic got better coverage. On cl100k_base, Hindi was closer to 4-6x English; on o200k_base it dropped to 2-3x. Japanese improved from ~3x to ~1.7x.
The improvement is uneven. o200k_base focused on high-volume non-English languages (Hindi, Japanese, Mandarin, Arabic). Low-resource languages like Swahili, Tamil, or Vietnamese saw smaller improvements.
**Always validate.** Run tiktoken on representative samples before pricing a multilingual deployment; the ratios above are guides, not billing primitives.
**Approximate ratios on o200k_base:**
- English: 1.0 (baseline)
- Spanish, French, Italian, Portuguese: 1.1-1.3 (Latin-script overlap)
- German: 1.3-1.5 (compound words)
- Mandarin: 1.5
- Japanese: 1.5-1.7
- Arabic: 2.0
- Hindi (Devanagari): 2-3
- Korean: 2.0
- Vietnamese, Tamil, Bengali: 2.5-3.5 (low-resource scripts)
**Why o200k_base improved on cl100k_base.** OpenAI doubled the vocabulary from 100K to 200K and explicitly allocated the extra capacity to multilingual coverage. Concrete additions: dedicated Devanagari tokens for common Hindi syllables, dedicated CJK tokens for frequent kanji and full hiragana/katakana coverage, expanded Arabic coverage, more Cyrillic. The result: Hindi dropped from a ~4-6x penalty on cl100k_base to ~2-3x on o200k_base. Japanese improved from ~3x to ~1.7x. Mandarin from ~2.5x to ~1.5x.
**The improvement is uneven.** OpenAI optimized for languages with high web volume and large speaker counts. Hindi, Japanese, Mandarin, and Arabic saw the biggest gains. Lower-resource languages saw smaller improvements, because the BPE training corpus had less of them to merge from. Tamil, Telugu, and Bengali are still in the 2.5-3.5x range.
**Production implications.** Per-language fertility translates directly to cost. A 200-word Hindi message costs 2-3x what the same content in English would cost on input tokens. Cost models for multilingual deployments need a per-language multiplier, not a single global tokens/word ratio. Validate with `tiktoken.get_encoding('o200k_base')` on real production samples; the ratios above are guides, and content type (formal vs informal, code-mixed languages) shifts them meaningfully.
Quick reference
TL;DR
On o200k_base: English 1.0, Spanish ~1.2, Hindi 2-3, Japanese 1.5-2. Better than cl100k_base because o200k_base added dedicated CJK and Devanagari tokens. Validate with tiktoken on real samples.
Imagine four people from different countries trying to send the same message using a shared shorthand notebook. The notebook was designed mostly with English in mind. The English speaker writes the message in a few quick squiggles. The Spanish speaker is also pretty fast because their alphabet is similar. The Hindi and Japanese speakers find that their writing systems are not really in the notebook, so they have to spell things out in tiny pieces, which takes much more space. A newer notebook (o200k_base) added shortcuts for Hindi, Japanese, and Chinese characters, so they now spell things out in fewer pieces than before, but they are still slower than English. The numbers: English baseline of 1, Spanish about 1.2, Hindi about 2.5, Japanese about 1.7. These ratios determine how much you pay per message in each language.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Pricing an English-monolingual deployment with no multilingual users. Per-language ratios add complexity without benefit; a single English-tuned cost model is fine when traffic is single-language.
Using a model with a different tokenizer family like Llama 3 or Qwen, where fertility ratios differ from o200k_base. The ratios above are specific to OpenAI's o200k_base; other tokenizers have their own per-language profiles.
Estimating costs for Google Gemini, which bills per character rather than per token. Token-based cost modeling is the wrong unit for Gemini; character-count drives the bill and decouples from tokenizer fertility.
Quick scan · 2–4 min
Everything important, quickly.
Lead with the ratios: English 1.0, Spanish ~1.2, Hindi 2-3, Japanese 1.5-2 on o200k_base. Explain that o200k_base doubled vocabulary to 200K and allocated heavily to multilingual coverage (dedicated Devanagari and CJK tokens). Quote the improvement vs cl100k_base (Hindi halved, Japanese dropped from 3x to ~1.7x). Cover the operational implications: per-language cost multipliers, context-window math, the fairness angle. Close with validation discipline: tiktoken on real samples per language.
Real-world examples
Real products, models, and research that use this idea.
1tiktoken.get_encoding('o200k_base') on Hindi medical text in 2026 returns roughly 2.5 tokens per word versus 1.3 for English on the same content domain.
2OpenAI's GPT-4o tokenizer announcement explicitly highlighted improved multilingual fertility as a design goal, citing CJK and Devanagari as primary targets.
3Anthropic's Claude family tokenizer is not published but empirically shows similar non-English fertility patterns; running messages.count_tokens on multilingual samples reveals the per-language cost.
4Llama 3+ and Qwen 3.5 ship with vocabularies designed for stronger non-English coverage than OpenAI's tokenizers; teams with heavy Hindi or Mandarin workloads sometimes route to those open-weight models for cost reasons.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QYour service expands to Tamil and Bengali users. What do you expect to happen to per-ticket cost?
Sign in or go Pro to see the approach for this follow-up.
QHow would code-mixed content (Hinglish: Hindi+English in the same message) tokenize compared to pure Hindi?
Sign in or go Pro to see the approach for this follow-up.
QWhy does o200k_base improve Japanese more than it improves Korean, despite both being CJK?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming a single 'tokens per word' multiplier applies across all languages, when fertility varies 1.0 to 3.0 within o200k_base alone.