How does tokenizer fertility bias affect cost and context window budget for non-English API users?

The startup's problem is that the GPT-4 API charges per token, and a token is not the same as a word. GPT-4's tokenizer, cl100k_base, was trained mostly on English, so common English words became single tokens. Arabic is handled poorly by that same tokenizer. Each Arabic character takes 2 bytes in UTF-8, the standard text encoding, and the tokenizer learned almost no rules for joining Arabic pieces. So Arabic words break into 3 to 5 small tokens each instead of 1. That means the 800-token English conversation becomes roughly 2,400 to 4,000 tokens in Arabic for the same meaning. This phenomenon is called **fertility**, the number of tokens per word. The consequence is threefold. The bill is 3 to 5 times higher, the model can hold 3 to 5 times less conversation history in the same window, and the per-minute token limit fills up faster. The right move is to tokenize real Arabic conversations and measure the true ratio before setting any budget.

GPT-4 uses `cl100k_base`, a byte-level BPE tokenizer trained on an English-heavy corpus, so English tokenizes near 1 token per word. Arabic does not get that deal. Arabic codepoints are 2 bytes in UTF-8, and the merge table holds few Arabic pairs, so Arabic words fragment into 3-5 tokens each. That is the **fertility** multiplier. $$\text{Arabic tokens} \approx \text{fertility} \times \text{English tokens}$$ The 800-token English conversation becomes roughly 2,400-4,000 Arabic tokens. Three budgets degrade together, and a team that extrapolates from the English prototype will get all three wrong. Cost rises 3-5x per conversation. The effective context window shrinks by the same factor, so history and retrieved context must be trimmed harder. Tokens-per-minute rate limits are consumed faster, throttling throughput before request-count limits bind. The correct engineering move is not to guess the multiplier. Sample around 100 real Arabic conversations, tokenize them with `tiktoken`, measure the actual fertility, and use that measured number for cost forecasting, context-budget sizing, and rate-limit headroom.