How would you build an accurate token budget estimator for a multilingual RAG application?
A product manager claims 'Our 128k context model can hold a 128,000-word document.' You are building a multilingual RAG pipeline serving English, French, Japanese, and Arabic users. Correct the PM's claim for English and explain how token budgeting must differ for each language. Then outline a practical method for estimating per-language token budgets.
Context windows are measured in tokens, not words, and fertility varies 1.3x to 5x by language, so budget by measuring tokens per word per language with the real tokenizer, never from word counts.
Imagine a suitcase that holds a fixed number of folded shirts, not a fixed number of outfits. English outfits fold small, so many fit. Japanese and Arabic outfits fold bulkier, so fewer fit even though the suitcase is the same size. The suitcase is the context window, the shirts are tokens, and the outfits are words. The product manager assumed one word equals one shirt, but it does not: a word can take one shirt in English and three to five in Japanese. To pack right, you weigh real clothes per language instead of guessing.
Detailed answer & concept explanation~6 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
3 min: tokens not words + English ~1.3x correction + fertility varies by language + Japanese/Arabic 3-5x + sample real docs + per-language table + p95 buffer + rate limits.
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o") # o200k_base
def fertility(samples: list[str]) -> float:
total_tokens = sum(len(enc.encode(doc)) for doc in samples)
total_words = sum(len(doc.split()) for doc in samples)
return total_tokens / max(total_words, 1)
# Build a per-language table from real sampled documents
budget_table = {
lang: fertility(docs) for lang, docs in samples_by_language.items()
}
# At request time, count tokens directly (never estimate from words)
def fits(doc: str, window: int = 128_000) -> bool:
return len(enc.encode(doc)) <= window| Language | Approx. fertility (tokens/word) | Words in a 128k window |
|---|---|---|
| English | ~1.3x | ~96,000 |
| French | ~1.1-1.5x | ~85,000-115,000 |
| Japanese | ~3-5x | ~25,000-40,000 |
| Arabic | ~3-5x | ~25,000-40,000 |
Real products, models, and research that use this idea.
- OpenAI bills GPT-5.5 by token, so a multilingual RAG product must size retrieval by tokens per language.
- Perplexity retrieves and stuffs documents into the context window, where high-fertility languages eat the budget faster.
- tiktoken is the canonical way to measure per-language fertility for OpenAI models before sizing a budget.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you handle documents that mix languages within a single chunk?
QHow does switching from cl100k_base to o200k_base change your budget table?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Telling the PM the model holds 128,000 words. Context windows count tokens, and English alone runs about 1.3 tokens per word, so the real figure is closer to 96,000 words.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.