BPE learns merge rules by frequency, so an English-heavy corpus gives few merges to underrepresented languages, fragmenting their words and costing 3-10x more tokens per meaning.
Imagine a shorthand notebook where you invent symbols for the phrases you write most often. You write mostly in English, so you build hundreds of English shortcuts and almost none for Swahili. Now you try to write a Swahili sentence. You have no shortcuts, so you spell every piece out letter by letter, and it takes many more strokes than the same idea in English. A tokenizer works the same way: it builds shortcuts, called merges, for the language it saw most. Languages it barely saw get spelled out into many small pieces, so the same meaning costs far more tokens. That extra cost is the token tax.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The token tax is not a bug or a deliberate penalty. It is the natural consequence of BPE's frequency-based learning applied to an imbalanced corpus. To answer this confidently you trace the mechanism from tokenizer training through inference to downstream accuracy.
The question rewards spotting that three of the four options are surface explanations. Word length, a language-detection penalty, and Unicode width all sound plausible but miss the real driver. Each one points at a property of the language or the encoding, when the actual cause is a property of the training corpus that built the tokenizer.
We will pin down how merges get allocated, define fertility so the disparity is measurable, show why byte-level BPE does not rescue low-resource languages, explain why the whole thing is frozen in at tokenizer-training time, and lay out the distractors so the misconception does not survive the explanation.
How merge rules get allocated
BPE training scans the corpus and repeatedly merges the most frequent adjacent pair of symbols into one new token. Do this tens of thousands of times and you get a vocabulary whose shortcuts mirror the statistics of the data.
When the corpus is dominated by English, the frequent pairs are English. Sequences like 'the', 'ing', 'tion', and 'ation' all earn merges and collapse to single tokens. A language present in a fraction of a percent of the data contributes almost nothing to the frequency counts, so it earns only a handful of merges.
The practical effect is that most words in an underrepresented language never get assembled into compact tokens. They stay near the byte floor, splitting into many small pieces. This is the option-one mechanism: fewer applicable merges, more tokens per word. It has nothing to do with how long the words are.
The subtlety worth internalizing is that merges are a shared, finite budget. A 100k vocabulary can hold only 100k tokens, and the frequency-driven process spends almost all of that budget on whatever dominates the corpus. So it is not that low-resource languages were singled out; they simply lost the competition for merge slots to the language that appeared most. Change the corpus mix and the same algorithm would allocate merges differently, which is the clearest evidence that the tax lives in the data, not the language.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
# Semantically similar sentences across languages
samples = {
"English": "The relationship between data and accuracy matters.",
"Spanish": "La relacion entre datos y precision importa.",
"Swahili": "Uhusiano kati ya data na usahihi ni muhimu.",
}
for lang, text in samples.items():
n_tok = len(enc.encode(text))
n_word = len(text.split())
print(lang, n_tok, "tokens", "fertility", round(n_tok / n_word, 2))Real products, models, and research that use this idea.
- Users running GPT-5.x or Claude Opus 4.7 over Swahili or Thai pay several times more per sentence than English users for the same meaning, because token counts inflate.
- Multilingual benchmarks like FLORES show accuracy gaps for high-fertility languages that track tokenizer imbalance rather than model size.
What an interviewer would ask next. Try answering before peeking at the approach.
QThe Token Tax work links doubling fertility to roughly 4x training cost for equal accuracy. What mechanism explains that?
Reason about semantic signal per token; high-fertility tokens are byte fragments, so the model needs far more tokens, and thus compute, to absorb the same patterns.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Blaming word length for the token tax. Token count is driven by how many merge rules the language got, not by inherently longer words.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.