What primarily causes the 'token tax' for low-resource languages in BPE tokenizers?

Attempt it

Quick reference

TL;DR

BPE learns merge rules by frequency, so an English-heavy corpus gives few merges to underrepresented languages, fragmenting their words and costing 3-10x more tokens per meaning.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a shorthand notebook where you invent symbols for the phrases you write most often. You write mostly in English, so you build hundreds of English shortcuts and almost none for Swahili. Now you try to write a Swahili sentence. You have no shortcuts, so you spell every piece out letter by letter, and it takes many more strokes than the same idea in English. A tokenizer works the same way: it builds shortcuts, called merges, for the language it saw most. Languages it barely saw get spelled out into many small pieces, so the same meaning costs far more tokens. That extra cost is the token tax.

Key concepts

BPE merge rules Token tax Fertility (tokens per word)Training corpus imbalance

Quick scan · 2–4 min

Everything important, quickly.

3 min: merges allocated by frequency + fertility metric + byte-level does not fix it + three downstream costs + tokenizer-time lock-in.

Code example

python

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

# Semantically similar sentences across languages
samples = {
    "English": "The relationship between data and accuracy matters.",
    "Spanish": "La relacion entre datos y precision importa.",
    "Swahili": "Uhusiano kati ya data na usahihi ni muhimu.",
}
for lang, text in samples.items():
    n_tok = len(enc.encode(text))
    n_word = len(text.split())
    print(lang, n_tok, "tokens", "fertility", round(n_tok / n_word, 2))

Real-world examples

Real products, models, and research that use this idea.

Users running GPT-5.x or Claude Opus 4.7 over Swahili or Thai pay several times more per sentence than English users for the same meaning, because token counts inflate.
Multilingual benchmarks like FLORES show accuracy gaps for high-fertility languages that track tokenizer imbalance rather than model size.
Meta's Llama 4 and Google's Gemini 3 invest in more multilingual tokenizer coverage specifically to shrink the fertility gap for non-English scripts.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QThe Token Tax work links doubling fertility to roughly 4x training cost for equal accuracy. What mechanism explains that?

Sign in or go Pro to see the approach for this follow-up.

QHow would you design tokenizer training to shrink the fertility gap for a target low-resource language?

Sign in or go Pro to see the approach for this follow-up.

QWhen would STRR show a language as fair while fertility flags it as penalized?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.

The Token Tax: Measuring Tokenization Bias and Its Effect on Language Model Performance