Take one short sentence and its translations: the sample OpenAI used to announce the GPT-4o tokenizer, "Hello, my name is GPT-4o. I'm a new type of language model, it's nice to meet you!", in English, Spanish, Hindi, and Japanese. Tokenize each version with o200k_base. Estimate the ratio of token counts for each language relative to English (English baseline = 1.0). Then briefly explain how and why these ratios improved on o200k_base versus cl100k_base.
On OpenAI's sample sentence, o200k_base gives English 1.0, Spanish ~1.1, Hindi ~1.3, Japanese ~1.1 (Hindi was ~3.3 on cl100k_base). Real text varies; validate with tiktoken.
Imagine four people from different countries trying to send the same message using a shared shorthand notebook. The notebook was designed mostly with English in mind. The English speaker writes the message in a few quick squiggles. The Spanish speaker is also pretty fast because their alphabet is similar. The Hindi and Japanese speakers find that their writing systems are not really in the notebook. They have to spell things out in tiny pieces, which takes much more space. A newer notebook (o200k_base) added thousands of shortcuts for Hindi, Japanese, and Chinese characters, so they now spell things out in far fewer pieces, close to English. On OpenAI's sample sentence: English baseline of 1, Spanish about 1.1, Hindi about 1.3, Japanese about 1.1. Longer real messages can show bigger gaps. These ratios determine how much you pay per message in each language.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Tokenizer fertility per language directly determines per user cost and per user context budget in any multilingual deployment. On OpenAI's sample sentence, o200k_base gives English 1.0, Spanish about 1.1, Japanese about 1.1, and Hindi about 1.3; real text varies.
Writing system as the primary driver of fertility
Two things determine tokenizer fertility for a language. The first is how the BPE merge table allocated tokens to patterns common in that language. The second is how the language's writing system maps to UTF-8 bytes. Both factors compound.
The figures below come from OpenAI's GPT-4o sample sentence (English: 27 tokens on cl100k_base, 24 on o200k_base). One short sentence is a small sample, so treat them as direction, not as billing figures.
Latin-script languages (English, Spanish, French, Italian, German, Portuguese) share the ASCII range plus a small set of accented characters. The byte representation is 1-2 bytes per character. BPE merge tables learned merges that cover Latin patterns broadly. Spanish or French content reuses many of the same tokens as English, with small per language additions. Result: fertility close to English, about 1.1-1.2x on the sample.
German sits at the upper end of Latin (29 tokens versus 24, about 1.2x). On longer text, compound words like 'Donaudampfschifffahrtsgesellschaftskapitän' do not have dedicated tokens and fragment into pieces.
CJK (Chinese, Japanese, Korean) uses ideographs and syllabaries that are 3 bytes per character in UTF-8 for common characters. Without dedicated merges, each character splits into up to 3 byte tokens. o200k_base holds thousands of multi-character kana and Han character tokens (about 4,600, versus about 300 in cl100k_base). The sample takes 24 tokens in Chinese, 26 in Japanese, and 27 in Korean, all close to English. Kanji share Unicode code points with Chinese characters, so one token can serve both languages.
Devanagari (Hindi, Marathi, Nepali) and other Indic scripts (Gujarati, Tamil, Telugu, Bengali) are 3 bytes per character in UTF-8. Indic scripts also have complex grapheme clusters where multiple code points combine into a single visual character. cl100k_base has almost no multi-character Devanagari tokens, so the Hindi sample took 90 tokens (about 3.3x English). o200k_base has about 3,800 such tokens, and the Hindi sample takes 31 (about 1.3x). Other Indic scripts gained even more: Gujarati went from 145 to 33 tokens, Telugu from 159 to 45, and Tamil from 116 to 35.
Arabic and Hebrew are 2 bytes per character in UTF-8 and have complex morphology (root-based word formation) that fragments under BPE without language-specific design. The Arabic sample dropped from 53 tokens to 26 (about 1.1x English). Hebrew is not in OpenAI's table.
Vietnamese uses a Latin-derived alphabet with extensive diacritics, which fragment under naive BPE. The Vietnamese sample takes 30 tokens on o200k_base (about 1.25x English), down from 46.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- tiktoken.get_encoding('o200k_base') returns 31 tokens for the Hindi version of OpenAI's GPT-4o sample sentence versus 24 for English; cl100k_base needs 90 tokens for the same Hindi sentence.
- OpenAI's GPT-4o announcement lists token counts for one sample sentence in 20 languages on the old and new tokenizers; the largest reductions were for Gujarati (4.4x), Telugu (3.5x), and Tamil (3.3x).
What an interviewer would ask next. Try answering before peeking at the approach.
QYour service expands to Tamil and Bengali users. What do you expect to happen to per-ticket cost?
Measure first. Tamil was among the biggest gainers on o200k_base (116 to 35 tokens on OpenAI's sample sentence, about 1.5x English). Bengali is not in OpenAI's table, so run tiktoken on real Tamil and Bengali tickets. Expect per-ticket input cost to rise above the English baseline, and by more on long technical text than on short sentences. Mitigations: compare other models' tokenizers on the same samples (including specialized Indic models like Sarvam), or extend the vocabulary of an open-weight model you fine-tune yourself.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a single 'tokens per word' multiplier applies across all languages, when OpenAI's sample sentence alone ranges from about 1.0x to 1.9x English across 20 languages on o200k_base.
60 second bullets to scan on the way to the call.
Approximate o200k_base fertility ratios for English, Spanish, Hindi, Japanese.
Why Latin-script languages tokenize close to English.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.