A startup is building a customer service chatbot for Arabic-speaking users using the GPT-4 API. Their English-language prototypes averaged 800 tokens per conversation. Explain why their Arabic conversations will cost significantly more, quantify the expected token ratio, and describe how they should adjust their cost and rate-limit planning.
Free · 2 AI evals / day
The startup's problem is that the GPT-4 API charges per token, and a token is not the same as a word. GPT-4's tokenizer, cl100k_base, was trained mostly on English, so common English words became single tokens.
Arabic is handled poorly by that same tokenizer. Each Arabic character takes 2 bytes in UTF-8, the standard text encoding, and the tokenizer learned almost no rules for joining Arabic pieces. So Arabic words break into 3 to 5 small tokens each instead of 1.
That means the 800-token English conversation becomes roughly 2,400 to 4,000 tokens in Arabic for the same meaning. This phenomenon is called **fertility**, the number of tokens per word.
The consequence is threefold. The bill is 3 to 5 times higher, the model can hold 3 to 5 times less conversation history in the same window, and the per-minute token limit fills up faster. The right move is to tokenize real Arabic conversations and measure the true ratio before setting any budget.
GPT-4 uses `cl100k_base`, a byte-level BPE tokenizer trained on an English-heavy corpus, so English tokenizes near 1 token per word. Arabic does not get that deal.
Arabic codepoints are 2 bytes in UTF-8, and the merge table holds few Arabic pairs, so Arabic words fragment into 3-5 tokens each. That is the **fertility** multiplier.
$$\text{Arabic tokens} \approx \text{fertility} \times \text{English tokens}$$
The 800-token English conversation becomes roughly 2,400-4,000 Arabic tokens.
Three budgets degrade together, and a team that extrapolates from the English prototype will get all three wrong. Cost rises 3-5x per conversation. The effective context window shrinks by the same factor, so history and retrieved context must be trimmed harder. Tokens-per-minute rate limits are consumed faster, throttling throughput before request-count limits bind.
The correct engineering move is not to guess the multiplier. Sample around 100 real Arabic conversations, tokenize them with `tiktoken`, measure the actual fertility, and use that measured number for cost forecasting, context-budget sizing, and rate-limit headroom.
Quick reference
TL;DR
Arabic fragments into 3-5x more tokens than English under cl100k_base, so cost, context window, and rate limits all degrade by the same multiplier.
Imagine you bought movers who charge by the box. English furniture comes pre-packed into a few big crates, so two rooms fit in two crates. Arabic furniture arrives unpacked, piece by piece, so the same two rooms need ten small boxes. The movers charge per box, the truck fills up faster, and you can carry fewer rooms per trip.
The tokenizer is the mover, and tokens are the boxes. English packs into few tokens; Arabic spreads into many. Same conversation, but you pay more, you fit less history in one trip, and you hit the truck's limit sooner. The fix is to count the boxes on a real Arabic shipment before you book the move.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Forecasting cost for a product whose traffic is almost entirely English Fertility is near 1 there, so the multiplier analysis adds nothing and word count is already a fine proxy.
Locking a budget before any real Arabic conversation data exists The true multiplier varies by dialect, topic, and tokenizer, so a number set without measurement will be off.
Carrying a fertility figure from one model to another with a different tokenizer o200k_base and cl100k_base fragment Arabic differently, so the measured ratio does not transfer across encodings.
Quick scan · 2–4 min
Everything important, quickly.
2 min: English-biased merges + Arabic fertility 3-5x + three token budgets (cost, context, rate limit) + measure with tiktoken on real data.
Code example
python
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
# Sample real conversations instead of guessing the multiplier.
en_convos = load_sample("english", n=100)
ar_convos = load_sample("arabic", n=100)
def avg_tokens(convos):
return sum(len(enc.encode(c)) for c in convos) / len(convos)
en_avg = avg_tokens(en_convos) # ~800
ar_avg = avg_tokens(ar_convos) # ~2400-4000
fertility = ar_avg / en_avg # measured multiplier for forecasting
print(en_avg, ar_avg, fertility)
Real-world examples
Real products, models, and research that use this idea.
1Teams shipping Arabic or Hindi GPT-5.5 support bots routinely budget 3-5x the English token cost per conversation after measuring fertility with tiktoken.
2Multilingual RAG products tune chunk size per language because a 128k context window holds far less Arabic than English content.
3Providers like OpenAI and Cohere document tokens per minute rate limits, which non-Latin deployments hit faster despite identical request volume.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you keep more conversation history in context when Arabic fertility eats your window?
Sign in or go Pro to see the approach for this follow-up.
QIf cost is unacceptable, when would you fine-tune or switch tokenizers instead of paying the tax?
Sign in or go Pro to see the approach for this follow-up.
QHow do you set provisioned throughput so an Arabic launch does not throttle at peak?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Extrapolating the Arabic budget straight from the 800-token English prototype, then discovering cost, context, and rate limits are all 3-5x worse in production.