Describe the 'token tax' phenomenon in multilingual LLMs. Explain the mechanism by which English centric BPE training creates systematic accuracy disparities, and cite the empirical evidence linking fertility to accuracy differences.
The token tax is the fertility gap between English and low-resource languages from BPE merge scarcity; empirically, doubling fertility needs about 4x more training compute for equal accuracy.
Imagine a timed reading test where English readers get flash cards for whole common words, so 'the' is one flip. Swahili readers got no cards, so they must spell every word out letter by letter. The clock counts flips, not letters. So the Swahili readers run out of time on the same passage even though the ideas are identical. A tokenizer hands out those flash cards, called merges, mostly to the language it studied most, usually English. Other languages get spelled out into many tiny pieces, so they burn through the budget faster and the model gets less practice with them. That double penalty, in cost and in learning, is the token tax.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The token tax is the structural accuracy disadvantage baked into BPE tokenizers trained on imbalanced multilingual corpora. It operates through fertility, the ratio of tokens to words, which diverges sharply across languages depending on their share of the tokenizer's training data.
To answer this at a senior level you cannot stop at cost. The interviewer is listening for whether you connect a tokenizer-training statistic to a model-accuracy outcome, which is the leap that separates a surface answer from a real one. You have to trace the full causal chain, from BPE training statistics, through merge-rule allocation, to pretraining data efficiency, and finally to the downstream accuracy gap that the empirical work measures.
We will build that chain in order, attach the one number worth memorizing, cover the production consequences that make it concrete, and close on why the bias is locked in before the model trains and what that constrains about mitigation.
Merge-rule allocation sets fertility
BPE training counts byte-pair frequencies across the corpus and repeatedly merges the most frequent pair, iterating until it hits the target vocabulary size. It is effectively a greedy compression algorithm fit to the data it saw.
Languages with a large corpus share saturate the high-frequency positions and earn many productive merges. Underrepresented languages contribute little to the counts and earn few, so their words fragment toward the byte floor.
The per-language result is captured by fertility:
In cl100k_base, English lands near 1.3. Spanish and other Latin-script European languages stay close, because they share script and many subword patterns with English and thus benefit from the same merges. Burmese and similar low-resource non-Latin scripts can exceed 7. The same 1000-word document is about 1300 tokens in English and 8000 in Burmese, a 6x overhead that has nothing to do with the language being inherently more complex.
The phrase measurement artifact is worth dwelling on. Burmese is not harder for a model to represent in principle; its words are not meaningfully longer or denser. The inflation comes entirely from the tokenizer having seen too little of it to learn compact tokens. Run the same algorithm on a Burmese-heavy corpus and Burmese fertility would drop while English rose. The number reflects who trained the tokenizer, not the language.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
# Semantically similar sentences across languages
samples = {
"English": "Training data and model accuracy matter for multilingual systems.",
"Spanish": "Los datos y la precision del modelo importan en sistemas multilingues.",
"Swahili": "Data ya mafunzo na usahihi wa mfano ni muhimu kwa mifumo ya lugha nyingi.",
}
baseline = None
for lang, text in samples.items():
n_tok = len(enc.encode(text))
n_word = len(text.split())
baseline = baseline or n_tok
print(lang, n_tok, "tokens", "fertility", round(n_tok / n_word, 2),
"rel", round(n_tok / baseline, 1))Real products, models, and research that use this idea.
- Thai users of GPT-5.x pay several times more per query than English users, since Thai tokenizes at roughly 4x the token count for equivalent meaning.
- Amharic legal documents at high fertility exhaust a 128k window with content that fits in a fraction of that in English, forcing aggressive chunking.
What an interviewer would ask next. Try answering before peeking at the approach.
QThe 2x fertility to 4x compute relationship implies a power law. What exponent fits, and what does it imply for bringing a language at 8x fertility to parity?
Read 2 to 4 as an exponent of 2; apply it to a fertility ratio of 8 to get a 64x compute estimate and discuss why that is infeasible.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Framing the token tax as just a billing inconvenience. The link from fertility to accuracy makes it a structural capability gap, not only a cost difference.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.