Describe the 'token tax' phenomenon in multilingual LLMs. Explain the mechanism by which English centric BPE training creates systematic accuracy disparities, and cite the empirical evidence linking fertility to accuracy differences.
The token tax is the fertility gap between English and low-resource languages from BPE merge scarcity; higher fertility predicts lower accuracy, and doubling tokens can roughly quadruple training cost.
Imagine a timed reading test where English readers get flash cards for whole common words, so 'the' is one flip. Swahili readers got no cards, so they must spell every word out letter by letter. The clock counts flips, not letters. So the Swahili readers run out of time on the same passage even though the ideas are identical. A tokenizer hands out those flash cards, called merges, mostly to the language it studied most, usually English. Other languages get spelled out into many tiny pieces, so they burn through the budget faster and the model gets less practice with them. That double penalty, in cost and in learning, is the token tax.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The token tax is the structural accuracy disadvantage baked into BPE tokenizers trained on imbalanced multilingual corpora. It operates through fertility, the ratio of tokens to words, which diverges sharply across languages depending on their share of the tokenizer's training data.
The token tax goes beyond cost. A statistic from tokenizer training drives an outcome in model accuracy. The full causal chain runs from BPE training statistics, through the allocation of merge rules, to pretraining data efficiency. It ends at the downstream accuracy gap that the empirical work measures.
Merge rule allocation sets fertility
BPE training counts byte-pair frequencies across the corpus and repeatedly merges the most frequent pair, iterating until it hits the target vocabulary size. It is a greedy compression algorithm fit to the data it saw.
Languages with a large corpus share saturate the high-frequency positions and earn many productive merges. Underrepresented languages contribute little to the counts and earn few, so their words fragment toward the byte floor.
The result for each language is captured by fertility:
In cl100k_base, English lands near 1.3. Spanish and other Latin-script European languages stay close. They share script and many subword patterns with English, so they benefit from the same merges. Burmese and similar low-resource non-Latin scripts can exceed 7. The same 1000-word document is about 1300 tokens in English and 8000 in Burmese. That 6x overhead has nothing to do with the language being more complex.
This is a measurement artifact. In principle, Burmese is not harder for a model to represent. Its words are not much longer or denser. The inflation comes entirely from the tokenizer having seen too little of it to learn compact tokens. Run the same algorithm on a mostly Burmese corpus and Burmese fertility would drop while English rose. The number reflects who trained the tokenizer, not the language.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
# Semantically similar sentences across languages
samples = {
"English": "Training data and model accuracy matter for multilingual systems.",
"Spanish": "Los datos y la precision del modelo importan en sistemas multilingues.",
"Swahili": "Data ya mafunzo na usahihi wa mfano ni muhimu kwa mifumo ya lugha nyingi.",
}
baseline = None
for lang, text in samples.items():
n_tok = len(enc.encode(text))
n_word = len(text.split())
baseline = baseline or n_tok
print(lang, n_tok, "tokens", "fertility", round(n_tok / n_word, 2),
"rel", round(n_tok / baseline, 1))Real products, models, and research that use this idea.
- On cl100k_base (GPT-4), a Thai sentence takes two to five times the tokens of the same English sentence; o200k_base (GPT-4o, GPT-5) cuts that to between parity and about twice.
- Amharic legal documents at high fertility exhaust a 128k window with content that fits in a fraction of that in English, forcing aggressive chunking.
What an interviewer would ask next. Try answering before peeking at the approach.
QThe Token Tax paper estimates that doubling the token count roughly quadruples training cost. Where does the factor of four come from, and why is it an upper-end figure?
Attention cost grows with the square of sequence length, so twice the tokens gives about four times the attention compute; feed-forward layers grow only linearly, so the real total is lower.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Framing the token tax as just a billing inconvenience. The link from fertility to accuracy makes it a structural capability gap, not only a cost difference.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.