Define 'fertility' as a tokenizer metric and explain why high fertility for a language is both a fairness and a model performance problem.
Define the fertility metric for tokenizer evaluation. Explain the two separate problems that high fertility causes: (1) an economic fairness problem for users and (2) a model quality problem in terms of training and inference. Cite the empirical relationship between fertility and downstream accuracy if known.
Fertility is average tokens per word; high fertility taxes non-English users on cost and context, and starves the model of signal so accuracy lags at fixed compute.
Imagine a vending machine that charges by the piece, and your favorite snack happens to come pre-broken into ten little crumbs while everyone else's comes whole. You pay ten times to get the same snack, and it barely fits in your bag. Worse, when you try to learn what the snack tastes like, you only ever get crumbs, so you never quite get the full flavor. Fertility is how broken-up a language gets. Some languages come whole, others come in crumbs, and the crumb languages get charged more and understood less.
Detailed answer & concept explanation~4 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: fertility definition + fairness harm (billing + context) + quality harm (training signal) + 4x-at-2x figure + why both harms are separate + tail-hiding averages + stickiness of the fix.
Real products, models, and research that use this idea.
- OpenAI moved from cl100k_base to o200k_base partly to cut multilingual fertility, lowering both per-token cost and the accuracy gap for non-English GPT-5.5 users.
- Google's Gemini 3.1 Pro and Gemma 4 ship large multilingual vocabularies aimed at reducing fertility on Indic, Arabic, and African scripts.
- The BLOOM and Aya multilingual projects trained balanced tokenizers specifically to avoid English-centric fragmentation of low-resource languages.
- Cohere's enterprise localization work reports fertility alongside accuracy when choosing or extending a tokenizer for a target market.
What an interviewer would ask next. Try answering before peeking at the approach.
QFertility is an average. How would you catch a tokenizer that is fine on common words but shreds named entities?
QYou inherit a deployed model with bad fertility on a target market. What are your options short of full retraining?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating high fertility as a billing inconvenience only, and missing that it also lowers the model's actual accuracy on that language at fixed training compute.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.