Using tiktoken with cl100k_base (GPT-4's tokenizer), compare the token count for:
- English: 'Hello world' (2 words, 11 characters)
- Arabic: 'مرحبا بالعالم' (same meaning: 'Hello world', 2 words, 13 Arabic characters)
Arabic characters use 2 bytes each in UTF-8. The cl100k_base tokenizer was trained predominantly on English data. Predict the token count for each string and the approximate ratio.
A tokenizer is the lookup table that turns text into the numbers a model reads. GPT-4 uses one called cl100k_base, and it was built mostly from English text.
Because English was so common during training, whole words like 'Hello' and 'world' each got their own single token. So 'Hello world' is about 2 tokens, almost one per word.
Arabic is different. Each Arabic letter takes 2 bytes to store in UTF-8, the standard text encoding, and the tokenizer learned very few shortcuts for joining Arabic pieces together. So 'مرحبا بالعالم', which means the same thing, gets chopped into many small pieces, roughly 8 to 10 tokens.
The punchline: identical meaning, but Arabic costs 4 to 5 times more tokens. Since APIs bill per token and the context window is counted in tokens, the Arabic user quietly pays a tax for the same sentence. This gap is called the **token tax**.
The trap here is assuming token count tracks meaning. It tracks the tokenizer's merge table, and `cl100k_base` was trained on a corpus dominated by English.
The phrase 'Hello world' lands at about 2 tokens because both words earned their own entries during training. The Arabic 'مرحبا بالعالم' lands at roughly 8-10 tokens, a 4-5x blowup for the same idea.
Two mechanisms stack. First, Arabic codepoints encode as 2 bytes each in UTF-8, so the byte sequence is already longer. Second, the merge table holds almost no Arabic byte pairs, so those bytes never combine into word-level tokens. The result is high **fertility**, the tokens per word ratio:
$$\text{fertility} = \frac{\#\,\text{tokens}}{\#\,\text{words}}$$
English fertility here is near 1; Arabic is around 4-5. The practical bite is threefold: API cost scales with tokens, the usable context window shrinks for the same content, and tokens per minute rate limits hit sooner. The honest interview move is to refuse a precise count and say 'measure it with tiktoken on real data', because the exact number depends on the specific strings.
Quick reference
TL;DR
English 'Hello world' is about 2 tokens; the Arabic equivalent is roughly 8-10, a 4-5x token tax driven by English-biased merge rules.
Imagine a vending machine stocked to dispense whole sandwiches for the most popular orders. Ask for a regular sandwich and one button gives you the whole thing. Order something the owner never expected and the machine has no sandwich button, so it hands you the ingredients one slice at a time, and you pay per slice.
That is what happens here. The tokenizer learned shortcuts for common English, so 'Hello' pops out as one piece. Arabic was rarely stocked, so its words come out almost letter by letter, and you pay for every little piece. Same meaning, many more pieces, a much bigger bill.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Estimating cost for an all-English product where every message is Latin-script Fertility sits near 1 there, so word count is a fine proxy and the token-tax analysis adds nothing.
Reporting a precise token count to a stakeholder before measuring real traffic The figure swings with the exact strings and tokenizer version, so a quoted integer will be wrong and misleading.
Comparing models whose tokenizers differ, like cl100k_base versus o200k_base Fertility is tokenizer-specific, so a ratio measured on one vocabulary does not carry over to another.
Real products, models, and research that use this idea.
1OpenAI bills GPT-5.5 calls per token via tiktoken, so an Arabic or Hindi chatbot pays several times more than an English one for equivalent messages.
2Cohere and other providers publish multilingual fertility studies showing 3-5x token inflation for Arabic, Hindi, and Thai under English-trained BPE vocabularies.
3Gemini 3.1 and Llama 4 expanded tokenizer vocabularies specifically to lower non-Latin fertility and narrow this cost gap for global deployments.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you forecast monthly API cost for a product that is 70% Arabic and 30% English traffic?
Sign in or go Pro to see the approach for this follow-up.
QWould switching from cl100k_base to o200k_base actually reduce the Arabic tax, and by how much?
Sign in or go Pro to see the approach for this follow-up.
QIf you controlled the tokenizer, how would you cut non-Latin fertility without retraining the whole model?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Estimating Arabic tokens from character count and assuming roughly 1 token per word, the way English behaves, then under-budgeting cost by 4-5x.