The same sentence in English, Spanish, Hindi, and Japanese. Predict the token ratio on o200k_base for each.

Predict output·Medium·4.0 · 0·~2 min·Relevant atAi4bharat Sarvam

Attempt it

Take the same semantic content (say a 100-word English passage) and translate it to Spanish, Hindi, and Japanese. Tokenize each translation with o200k_base. Predict the approximate ratio of token counts for each language relative to English (English baseline = 1.0). Then briefly explain why these ratios improved on o200k_base versus cl100k_base.

The ratios on o200k_base, the tokenizer for GPT-4o, GPT-5, and GPT-5.5: - English: 1.0 (baseline) - Spanish: ~1.1-1.3 - Hindi: ~2-3 - Japanese: ~1.5-2 The pattern follows the writing system. Spanish uses the Latin alphabet, so it shares most token coverage with English; a few extra tokens for accented characters and Spanish-specific patterns. Hindi uses Devanagari, an entirely different script with no Latin overlap; on older tokenizers Devanagari fell back to byte-level sequences. Japanese mixes kanji, hiragana, and katakana, three writing systems in one language. **Why the ratios improved on o200k_base versus cl100k_base.** OpenAI doubled the vocabulary from 100K to 200K when designing the GPT-4o tokenizer, and they explicitly allocated the extra 100K tokens to multilingual coverage. CJK (Chinese, Japanese, Korean) got dedicated multi-character tokens. Devanagari got dedicated tokens for common Hindi patterns. Arabic got better coverage. On cl100k_base, Hindi was closer to 4-6x English; on o200k_base it dropped to 2-3x. Japanese improved from ~3x to ~1.7x. The improvement is uneven. o200k_base focused on high-volume non-English languages (Hindi, Japanese, Mandarin, Arabic). Low-resource languages like Swahili, Tamil, or Vietnamese saw smaller improvements. **Always validate.** Run tiktoken on representative samples before pricing a multilingual deployment; the ratios above are guides, not billing primitives.

**Approximate ratios on o200k_base:** - English: 1.0 (baseline) - Spanish, French, Italian, Portuguese: 1.1-1.3 (Latin-script overlap) - German: 1.3-1.5 (compound words) - Mandarin: 1.5 - Japanese: 1.5-1.7 - Arabic: 2.0 - Hindi (Devanagari): 2-3 - Korean: 2.0 - Vietnamese, Tamil, Bengali: 2.5-3.5 (low-resource scripts) **Why o200k_base improved on cl100k_base.** OpenAI doubled the vocabulary from 100K to 200K and explicitly allocated the extra capacity to multilingual coverage. Concrete additions: dedicated Devanagari tokens for common Hindi syllables, dedicated CJK tokens for frequent kanji and full hiragana/katakana coverage, expanded Arabic coverage, more Cyrillic. The result: Hindi dropped from a ~4-6x penalty on cl100k_base to ~2-3x on o200k_base. Japanese improved from ~3x to ~1.7x. Mandarin from ~2.5x to ~1.5x. **The improvement is uneven.** OpenAI optimized for languages with high web volume and large speaker counts. Hindi, Japanese, Mandarin, and Arabic saw the biggest gains. Lower-resource languages saw smaller improvements, because the BPE training corpus had less of them to merge from. Tamil, Telugu, and Bengali are still in the 2.5-3.5x range. **Production implications.** Per-language fertility translates directly to cost. A 200-word Hindi message costs 2-3x what the same content in English would cost on input tokens. Cost models for multilingual deployments need a per-language multiplier, not a single global tokens/word ratio. Validate with `tiktoken.get_encoding('o200k_base')` on real production samples; the ratios above are guides, and content type (formal vs informal, code-mixed languages) shifts them meaningfully.

Quick reference

TL;DR

On o200k_base: English 1.0, Spanish ~1.2, Hindi 2-3, Japanese 1.5-2. Better than cl100k_base because o200k_base added dedicated CJK and Devanagari tokens. Validate with tiktoken on real samples.

Memory aid

Easy to grasp

Imagine four people from different countries trying to send the same message using a shared shorthand notebook. The notebook was designed mostly with English in mind. The English speaker writes the message in a few quick squiggles. The Spanish speaker is also pretty fast because their alphabet is similar. The Hindi and Japanese speakers find that their writing systems are not really in the notebook, so they have to spell things out in tiny pieces, which takes much more space. A newer notebook (o200k_base) added shortcuts for Hindi, Japanese, and Chinese characters, so they now spell things out in fewer pieces than before, but they are still slower than English. The numbers: English baseline of 1, Spanish about 1.2, Hindi about 2.5, Japanese about 1.7. These ratios determine how much you pay per message in each language.

Key concepts

Multilingual fertilityo200k_baseCJK tokens

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Lead with the ratios: English 1.0, Spanish ~1.2, Hindi 2-3, Japanese 1.5-2 on o200k_base. Explain that o200k_base doubled vocabulary to 200K and allocated heavily to multilingual coverage (dedicated Devanagari and CJK tokens). Quote the improvement vs cl100k_base (Hindi halved, Japanese dropped from 3x to ~1.7x). Cover the operational implications: per-language cost multipliers, context-window math, the fairness angle. Close with validation discipline: tiktoken on real samples per language.

Real-world examples

Real products, models, and research that use this idea.

tiktoken.get_encoding('o200k_base') on Hindi medical text in 2026 returns roughly 2.5 tokens per word versus 1.3 for English on the same content domain.
OpenAI's GPT-4o tokenizer announcement explicitly highlighted improved multilingual fertility as a design goal, citing CJK and Devanagari as primary targets.
Anthropic's Claude family tokenizer is not published but empirically shows similar non-English fertility patterns; running messages.count_tokens on multilingual samples reveals the per-language cost.
Llama 3+ and Qwen 3.5 ship with vocabularies designed for stronger non-English coverage than OpenAI's tokenizers; teams with heavy Hindi or Mandarin workloads sometimes route to those open-weight models for cost reasons.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QYour service expands to Tamil and Bengali users. What do you expect to happen to per-ticket cost?

QHow would code-mixed content (Hinglish: Hindi+English in the same message) tokenize compared to pure Hindi?

QWhy does o200k_base improve Japanese more than it improves Korean, despite both being CJK?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming a single 'tokens per word' multiplier applies across all languages, when fertility varies 1.0 to 3.0 within o200k_base alone.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.