Fertility of 2.5 means Hindi averages 2.5 tokens per word on cl100k_base. English scores around 1.3, so Hindi costs roughly 2x more per word across billing, context capacity, and latency.
Imagine the tokenizer is a translator paid per output word. An efficient translator uses one short symbol per common input word and finishes fast. A clumsy translator needs to spell things out letter by letter, taking more symbols and costing more money. Fertility measures how many output symbols the translator needs per input word. English on most tokenizers scores around 1.3, like a fluent translator. Hindi on cl100k_base scores 2.5, like a translator who knows only basic shortcuts. Tamil can hit 4 to 5 on the same tokenizer, which is why API calls in those languages cost more per character. Newer tokenizers with bigger vocabularies have improved these numbers, but the gap is still there.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Fertility is the metric you reach for when someone asks 'how much will it cost to process this Hindi corpus' or 'why does my Tamil workload burn through the context window so fast'. The number itself is simple: average tokens per word on a specific corpus under a specific tokenizer. The implications are large because almost everything in LLM economics is denominated in tokens.
This deep dive defines fertility precisely, explains why 2.5 for Hindi on cl100k_base is mediocre, walks the three operational consequences, characterizes the multilingual fertility gap, and explains how 2026 vocabulary expansion has narrowed but not closed it.
Fertility, defined and computed
Fertility is a tokenizer characterization metric: the average number of tokens a tokenizer emits per unit of input on a specified corpus. The unit is usually a word (tokens per word, TPW) or a fixed number of characters (tokens per 100 characters). Lower fertility means more compressed encoding of the measured content.
The computation is straightforward. Tokenize a representative corpus with the tokenizer of interest. Count total tokens. Divide by word count (or by characters divided by 100). The result is fertility on that corpus under that tokenizer.
Two subtleties matter. First, the word count is itself a tokenization decision. For European languages, whitespace splitting is usually adequate; for languages without whitespace word separation (Chinese, Japanese, Thai), a language-specific segmenter or a tokens-per-character metric avoids ambiguity.
Second, fertility is corpus dependent. The same tokenizer produces different fertility on Wikipedia text versus code versus chat logs. A meaningful fertility number always names both the tokenizer and the corpus.
So when someone says 'fertility is 2.5 for Hindi on cl100k_base', they mean: take representative Hindi text, run it through cl100k_base, and you get an average of 2.5 tokens per word. Compare that to English at around 1.3 on the same tokenizer, and the Hindi workload costs roughly twice as much per word in every token denominated dimension.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Ahia et al. 2023 published a 24-language fertility benchmark on cl100k_base showing English at 1.3 tokens per word and Telugu at around 7, a 5.5x cost gap between the two languages.
- OpenAI's jump from cl100k_base (100K vocab) to o200k_base (200K vocab) for GPT-4o and later models was explicitly motivated by improved multilingual and code fertility.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does English fertility differ between cl100k_base and o200k_base?
o200k_base has roughly twice the vocabulary, so more frequent English n-grams earn dedicated ids. English fertility drops from around 1.3 to around 1.2. The improvement is small because cl100k_base was already near optimal for English; the bigger gains from the larger vocabulary appear on non-English content.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing fertility with vocabulary size. A larger vocabulary tends to reduce fertility, but they measure different things: fertility is a realized compression ratio on a corpus, vocabulary size is a count of available tokens.
60 second bullets to scan on the way to the call.
Define fertility as tokens per word (or per 100 characters) on a specific corpus under a specific tokenizer.
Explain why lower fertility benefits cost, context capacity, and latency simultaneously.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.