Higher fertility predicts lower downstream accuracy across models, and doubling the token count is estimated to roughly quadruple training cost, so high-fertility languages lose on quality and cost.
Imagine two students copying the same story by hand. One writes in shorthand and finishes in 100 strokes; the other has to spell everything letter by letter and needs 400 strokes. Given the same amount of writing time, the shorthand student rereads and learns the story far better. A tokenizer is the shorthand. Languages that get chopped into many tiny tokens are like the slow writer. The model spends its limited budget on mechanical pieces instead of meaning, so it understands those languages less well.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The intuitive direction of this effect is the wrong one. It feels like splitting a word into more pieces should give a model more to work with, the way more pixels give a sharper image. For tokenizers, the opposite is true, and the gap it opens is one of the most important fairness issues in LLMs.
The effect rests on one definition and one empirical finding. The definition is fertility, the average number of tokens per word. The finding, from tokenizer fairness research, is that fertility reliably predicts downstream accuracy: higher fertility, lower accuracy. The same research estimates that doubling the token count roughly quadruples training cost and time.
What fertility measures and why the average matters
Fertility is deliberately simple: run a corpus through the tokenizer, count tokens, divide by word count.
English on a byte-level BPE vocabulary lands around 1.3. A well-supported European language might be 1.5 to 2. A language whose script or vocabulary was thin in the tokenizer's training data can hit 3, 4, or higher. Sometimes it breaks all the way down to individual bytes.
The number is an average, which both helps and hides. It helps because it is a single comparable figure across languages. It hides because the tail matters: rare words and named entities often fragment far worse than the average suggests. That is why fertility is usually reported alongside a retention metric. But as a first-order predictor of cost and quality, the average already tells you most of the story.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's o200k_base improved multilingual fertility over cl100k_base, narrowing the per-token cost gap for non-English GPT-4o users.
- Google's Gemma 3 uses a 262k-entry SentencePiece vocabulary shared with Gemini, built to cover many languages with fewer tokens per word.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf fertility predicts accuracy, can you just retrain the tokenizer on a balanced corpus and swap it into an existing model?
No; tokenizer choice is sticky. Changing it invalidates the embedding matrix, so you need at minimum embedding re-init plus continued pretraining, or full retraining.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming more tokens per word gives the model finer-grained signal and therefore better accuracy, when the empirical relationship runs the opposite way.
60 second bullets to scan on the way to the call.
Definition of fertility as tokens per word
Direction of the fertility versus accuracy relationship
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.