What is the 'token tax' and how does it create downstream accuracy disparities across languages?

Attempt it

Describe the 'token tax' phenomenon in multilingual LLMs. Explain the mechanism by which English-centric BPE training creates systematic accuracy disparities, and cite the empirical evidence linking fertility to accuracy differences.

Free · 2 AI evals / day

Quick reference

TL;DR

The token tax is the fertility gap between English and low-resource languages from BPE merge scarcity; empirically, doubling fertility needs about 4x more training compute for equal accuracy.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a timed reading test where English readers get flash cards for whole common words, so 'the' is one flip. Swahili readers got no cards, so they must spell every word out letter by letter. The clock counts flips, not letters. So the Swahili readers run out of time on the same passage even though the ideas are identical. A tokenizer hands out those flash cards, called merges, mostly to the language it studied most, usually English. Other languages get spelled out into many tiny pieces, so they burn through the budget faster and the model gets less practice with them. That double penalty, in cost and in learning, is the token tax.

Key concepts

Token tax Fertility (tokens per word)BPE merge rule allocation Effective training signal Multilingual corpus imbalance

Quick scan · 2–4 min

Everything important, quickly.

5 min: merge allocation by frequency + fertility metric + fixed budget into fewer words + near-quadratic compute law + three production costs + tokenizer-time lock-in.

Code example

python

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

# Semantically similar sentences across languages
samples = {
    "English": "Training data and model accuracy matter for multilingual systems.",
    "Spanish": "Los datos y la precision del modelo importan en sistemas multilingues.",
    "Swahili": "Data ya mafunzo na usahihi wa mfano ni muhimu kwa mifumo ya lugha nyingi.",
}
baseline = None
for lang, text in samples.items():
    n_tok = len(enc.encode(text))
    n_word = len(text.split())
    baseline = baseline or n_tok
    print(lang, n_tok, "tokens", "fertility", round(n_tok / n_word, 2),
          "rel", round(n_tok / baseline, 1))

Real-world examples

Real products, models, and research that use this idea.

Thai users of GPT-5.x pay several times more per query than English users, since Thai tokenizes at roughly 4x the token count for equivalent meaning.
Amharic legal documents at high fertility exhaust a 128k window with content that fits in a fraction of that in English, forcing aggressive chunking.
FLORES-200 and similar multilingual benchmarks show accuracy gaps that track fertility, which the Token Tax work formalizes across model families.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QThe 2x fertility to 4x compute relationship implies a power law. What exponent fits, and what does it imply for bringing a language at 8x fertility to parity?

Sign in or go Pro to see the approach for this follow-up.

QDesign a tokenizer training procedure that minimizes cross-language fertility variance at a fixed 100k vocabulary. What breaks?

Sign in or go Pro to see the approach for this follow-up.

QHow would you measure and compensate for the token tax inside a multilingual RAG pipeline?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.

The Token Tax: Measuring Tokenization Bias and Its Effect on Language Model Performance