Predict the approximate token count difference between 'Hello world' and its Arabic equivalent 'مرحبا بالعالم' in a GPT-4 tokenizer.

A tokenizer is the lookup table that turns text into the numbers a model reads. GPT-4 uses one called cl100k_base, and it was built mostly from English text. Because English was so common during training, whole words like 'Hello' and 'world' each got their own single token. So 'Hello world' is about 2 tokens, almost one per word. Arabic is different. Each Arabic letter takes 2 bytes to store in UTF-8, the standard text encoding, and the tokenizer learned very few shortcuts for joining Arabic pieces together. So 'مرحبا بالعالم', which means the same thing, gets chopped into many small pieces, roughly 8 to 10 tokens. The punchline: identical meaning, but Arabic costs 4 to 5 times more tokens. Since APIs bill per token and the context window is counted in tokens, the Arabic user quietly pays a tax for the same sentence. This gap is called the **token tax**.

The trap here is assuming token count tracks meaning. It tracks the tokenizer's merge table, and `cl100k_base` was trained on a corpus dominated by English. The phrase 'Hello world' lands at about 2 tokens because both words earned their own entries during training. The Arabic 'مرحبا بالعالم' lands at roughly 8-10 tokens, a 4-5x blowup for the same idea. Two mechanisms stack. First, Arabic codepoints encode as 2 bytes each in UTF-8, so the byte sequence is already longer. Second, the merge table holds almost no Arabic byte pairs, so those bytes never combine into word-level tokens. The result is high **fertility**, the tokens per word ratio: $$\text{fertility} = \frac{\#\,\text{tokens}}{\#\,\text{words}}$$ English fertility here is near 1; Arabic is around 4-5. The practical bite is threefold: API cost scales with tokens, the usable context window shrinks for the same content, and tokens per minute rate limits hit sooner. The honest interview move is to refuse a precise count and say 'measure it with tiktoken on real data', because the exact number depends on the specific strings.