What most inflates token count for European languages compared to English in a BPE tokenizer trained on English-dominant data?

A **BPE** tokenizer builds its vocabulary by merging the most frequent text pieces in its training data. If that data is mostly English, the merges it learns are mostly English: common words and word parts collapse into single tokens. European languages get less of this treatment for two reasons. First, accented characters like é, ü, and ñ appear far less in an English corpus, so they earn fewer dedicated merges. Second, these languages are morphologically rich, meaning they pack grammar into long word endings (verb conjugations like -ción, case endings like -heit). Those endings rarely got their own tokens. The consequence is that a Spanish or German word, with no single matching entry, gets rebuilt from several smaller pieces. The Spanish 'información' might be four tokens while the English 'information' is one or two, even though they mean the same thing. More tokens per word is called higher fertility, and it directly raises cost and uses up the context window faster.

The driver is sparse merge coverage, not the bytes themselves. Byte pair encoding learns its merges from the training corpus, so an English-dominant corpus produces a vocabulary tuned to English words and morphemes. Two properties of European languages then bite. Accented characters (é, ü, ñ) are comparatively rare in English data, so they get fewer dedicated merges. And these languages are morphologically rich, encoding grammar in long endings like -ción, -ment, or -heit that never earned their own tokens. When a word has no single matching entry, the encoder rebuilds it from smaller subunits. The result is higher fertility, more tokens per word. The Spanish 'información' may be four tokens against one or two for the English 'information'. The tempting wrong answer blames non-ASCII codepoints for always splitting in two, but modern byte-level BPE frequently has merges for common accented characters. The real cost comes from morphology and accent coverage, and since $$\text{cost} = \text{tokens} \times \text{price-per-1k}$$ higher fertility translates straight into a higher bill for the same meaning.