Higher fertility predicts lower downstream accuracy: doubling tokens per word correlates with roughly 4x more training cost to reach the same quality, so under-trained languages lag.
Imagine two students copying the same story by hand. One writes in shorthand and finishes in 100 strokes; the other has to spell everything letter by letter and needs 400 strokes. Given the same amount of writing time, the shorthand student rereads and learns the story far better. A tokenizer is the shorthand. Languages that get chopped into many tiny tokens are like the slow writer: the model spends its limited budget on mechanical pieces instead of meaning, so it understands those languages less well.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This question hinges on getting the direction of an effect right, and the wrong direction is the intuitive one. It feels like splitting a word into more pieces should give a model more to work with, the way more pixels give a sharper image. For tokenizers, the opposite is true, and the gap it opens is one of the most important fairness issues in modern LLMs.
The answer rests on one definition and one empirical finding. The definition is fertility, the average number of tokens per word. The finding, from tokenizer-fairness research, is that fertility predicts downstream accuracy, and steeply: doubling it tracks about four times the compute needed to keep accuracy level. We will build the intuition, name the mechanism on both the training and inference sides, and see why each distractor is a real-world misconception worth being able to refute.
What fertility measures and why the average matters
Fertility is deliberately simple: run a corpus through the tokenizer, count tokens, divide by word count.
English on a modern byte-level BPE vocabulary lands around 1.3. A well-supported European language might be 1.5 to 2. A language whose script or vocabulary was thin in the tokenizer's training data can hit 3, 4, or higher, sometimes fragmenting all the way down to individual bytes.
The number is an average, which both helps and hides. It helps because it is a single comparable figure across languages. It hides because the tail matters: rare words and named entities often fragment far worse than the average suggests. That is why fertility is usually reported alongside a retention metric, but as a first-order predictor of cost and quality, the average already tells you most of the story.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's o200k_base improved multilingual fertility over cl100k_base, narrowing the per-token cost and quality gap for non-English GPT-5.5 users.
- Google's Gemma 4 and Gemini 3.1 Pro use large multilingual vocabularies specifically to lower fertility on Indic and African scripts.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf fertility predicts accuracy, can you just retrain the tokenizer on a balanced corpus and swap it into an existing model?
No; tokenizer choice is sticky. Changing it invalidates the embedding matrix, so you need at minimum embedding re-init plus continued pretraining, or full retraining.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming more tokens per word gives the model finer-grained signal and therefore better accuracy, when the empirical relationship runs the opposite way.
60 second bullets to scan on the way to the call.
Definition of fertility as tokens per word
Direction of the fertility versus accuracy relationship
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.