What primarily causes the 'token tax' for low-resource languages in BPE tokenizers?

The correct answer is the first option: an English-dominant corpus gives BPE plenty of English **merge rules** and very few for low-resource languages, so their text fragments into many more tokens. Here is the mechanism. A BPE tokenizer learns to compress text by studying which character pairs appear most often in its training data. If that data is mostly English, it learns hundreds of shortcuts that turn common English words into single tokens. A language present in only a tiny slice of the data gets almost no shortcuts, so its words break down into individual characters or bytes. The other options are tempting but wrong. Word length is not the driver, there is no language-detection penalty in BPE, and high Unicode codepoints are a minor factor at most. The real cause is the imbalance in the training data, which is why the same idea can cost a low-resource language several times more tokens than English.

The answer is the first option, and the key idea is that BPE allocates merge rules proportional to frequency in its training corpus. When the corpus is dominated by English, BPE accumulates many merge rules for English patterns, so common English words compress to one or two tokens. A language with a tiny share of the corpus gets very few applicable merges, so its words stay fragmented down toward individual bytes. The same meaning then costs several times more tokens. The distractors each miss the mechanism. Word length is not the cause; a short word with no merges still fragments. There is no language-detection penalty in BPE; the model never identifies a language. And while high Unicode codepoints add bytes, the dominant factor is merge-rule scarcity, not the encoding width. This is why the token tax is structural. It is baked in when the tokenizer is trained on imbalanced data, and it shows up downstream as higher API cost, faster context-window exhaustion, and lower accuracy for those languages. The metric that captures it is fertility, the tokens produced per word.