Using tiktoken with cl100k_base (GPT-4's tokenizer), compare the token count for: - English: 'Hello world' (2 words, 11 characters) - Arabic: 'مرحبا بالعالم' (same meaning: 'Hello world', 2 words, 13 Arabic characters) Arabic characters use 2 bytes each in UTF-8. The cl100k_base tokenizer was trained predominantly on English data. Predict the token count for each string and the approximate ratio.
English 'Hello world' is about 2 tokens; the Arabic equivalent is roughly 8-10, a 4-5x token tax driven by English-biased merge rules.
Imagine a vending machine stocked to dispense whole sandwiches for the most popular orders. Ask for a regular sandwich and one button gives you the whole thing. Order something the owner never expected and the machine has no sandwich button, so it hands you the ingredients one slice at a time, and you pay per slice. That is what happens here. The tokenizer learned shortcuts for common English, so 'Hello' pops out as one piece. Arabic was rarely stocked, so its words come out almost letter by letter, and you pay for every little piece. Same meaning, many more pieces, a much bigger bill.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
On the surface this looks like a counting puzzle: how many tokens does an Arabic phrase take versus its English twin? It shows up in senior interviews because the answer exposes a quiet, structural bias in how language models read text. The cost is real money and real context window, and it lands disproportionately on non-English users.
The phrase 'مرحبا بالعالم' means exactly 'Hello world'. Same information, same two words. Yet GPT-4 spends several times more tokens on the Arabic. Understanding why forces you to connect three things that are easy to keep separate: how UTF-8 encodes characters, how byte-level BPE learns its merge table, and how token count maps onto cost, context, and rate limits.
We will build the answer in layers, then put real numbers on it so you can defend a range instead of bluffing a single integer.
Why 'Hello world' is about 2 tokens
GPT-4's tokenizer, cl100k_base, is a byte-level BPE tokenizer. It starts from the 256 possible byte values and repeatedly merges the most frequent adjacent pair in its training corpus, building a vocabulary of around 100k entries. The entries that win are the patterns that appeared most often, so the vocabulary is essentially a frequency snapshot of the training data.
That corpus was dominated by English. So the merge process kept gluing common English letter sequences together until entire frequent words became single tokens. 'Hello' and 'world' are both common enough to have earned their own dedicated token IDs, and the leading-space variant ' world' typically earns one too, because words are usually preceded by a space in running text.
The result is that English fertility, the tokens per word ratio, sits very close to 1. A two-word English phrase costs roughly two tokens. This is the baseline the rest of the analysis measures against, and it is exactly why people wrongly assume token count tracks word count in general.
It is worth being precise about what 'common' means here. The merges are greedy and frequency-ordered, so the tokenizer spends its limited vocabulary budget on whatever appeared most. An English-heavy corpus pushes that budget toward English, which is the structural root of every downstream effect we are about to trace.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import tiktoken
enc = tiktoken.get_encoding("cl100k_base") # GPT-4's tokenizer
english = "Hello world"
arabic = "مرحبا بالعالم" # same meaning
en_toks = enc.encode(english)
ar_toks = enc.encode(arabic)
print(len(en_toks)) # ~2
print(len(ar_toks)) # ~8-10
print(len(ar_toks) / len(en_toks)) # ~4-5x token tax| Property | English 'Hello world' | Arabic 'مرحبا بالعالم' |
|---|---|---|
| UTF-8 bytes per character | 1 byte (ASCII) | 2 bytes |
| Learned merge coverage | Extensive (word-level) | Sparse (few pairs merge) |
| Approx tokens | ~2 | ~8-10 |
| Fertility (tokens/word) | ~1 | ~4-5 |
| Relative cost / context use | Baseline | 4-5x higher |
Real products, models, and research that use this idea.
- OpenAI bills GPT-5.5 calls per token via tiktoken, so an Arabic or Hindi chatbot pays several times more than an English one for equivalent messages.
- Cohere and other providers publish multilingual fertility studies showing 3-5x token inflation for Arabic, Hindi, and Thai under English-trained BPE vocabularies.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you forecast monthly API cost for a product that is 70% Arabic and 30% English traffic?
Sample real conversations per language, measure fertility with tiktoken, then weight the per-language token totals by traffic share before applying price per token.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Estimating Arabic tokens from character count and assuming roughly 1 token per word, the way English behaves, then under-budgeting cost by 4-5x.
60 second bullets to scan on the way to the call.
Why English words tokenize near one token each in cl100k_base
UTF-8 byte width of Arabic codepoints and why it matters
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.