Define the fertility metric for tokenizer evaluation. Explain the two separate problems that high fertility causes: (1) an economic fairness problem for users and (2) a model quality problem in terms of training and inference. Cite the empirical relationship between fertility and downstream accuracy if known.
Fertility is average tokens per word; high fertility taxes non-English users on cost and context, and starves the model of signal so accuracy lags at fixed compute.
Imagine a vending machine that charges by the piece, and your favorite snack happens to come pre-broken into ten little crumbs while everyone else's comes whole. You pay ten times to get the same snack, and it barely fits in your bag. Worse, when you try to learn what the snack tastes like, you only ever get crumbs, so you never quite get the full flavor. Fertility is how broken-up a language gets. Some languages come whole, others come in crumbs, and the crumb languages get charged more and understood less.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Tokenizer fertility is one of those metrics that looks like an efficiency footnote and turns out to sit at the center of a fairness debate about who LLMs serve well. A strong answer to this question does two things: it defines fertility precisely, and it resists the urge to merge the two harms into one. They are genuinely separate, with different mechanisms and different remedies.
The definition is a ratio. The harms are an economic one borne by users and a quality one borne by the model. The empirical glue is a steep relationship between fertility and accuracy. We will take each in turn, attach a number where the research gives one, and end on why the whole thing is hard to fix after the fact, which is the part interviewers use to separate book knowledge from production experience.
Defining fertility precisely
Fertility is the average number of tokens a tokenizer emits per word, measured over a representative corpus:
Fertility 1.0 is the ideal of one token per word. English on a modern byte-level BPE vocabulary lands near 1.3. Languages whose scripts or vocabularies were thin in the tokenizer's training data routinely sit at 3 to 5, and in the worst cases fragment toward individual bytes.
The precision that matters in an interview is two-fold. First, it is per word, not per character or per sentence, so you need a word-segmentation convention. Second, it is an average over a corpus, which means it can hide a heavy tail. A tokenizer can post a respectable mean fertility while shredding rare words and named entities, which is exactly the content that carries the most meaning. That is why fertility is usually reported with a companion retention metric rather than alone.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI moved from cl100k_base to o200k_base partly to cut multilingual fertility, lowering both per-token cost and the accuracy gap for non-English GPT-5.5 users.
- Google's Gemini 3.1 Pro and Gemma 4 ship large multilingual vocabularies aimed at reducing fertility on Indic, Arabic, and African scripts.
What an interviewer would ask next. Try answering before peeking at the approach.
QFertility is an average. How would you catch a tokenizer that is fine on common words but shreds named entities?
Report per-domain fertility plus a retention metric like STRR; inspect the tail distribution rather than the mean.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating high fertility as a billing inconvenience only, and missing that it also lowers the model's actual accuracy on that language at fixed training compute.
60 second bullets to scan on the way to the call.
Precise definition of fertility as tokens per word over a corpus
The economic fairness harm: per-token billing and context cost
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.