You hear 'fertility is 2.5 for Hindi on cl100k_base'. What does that number mean and is it good?

Flashcard·Easy·4.0 · 0·~30s·Relevant atAi4bharat Sarvam

Attempt it

It means Hindi text averages 2.5 tokens per word under cl100k_base, and no, that is not good. English on the same tokenizer scores around 1.3, so Hindi costs roughly twice as much per word in API billing, context capacity, and generation speed. Newer tokenizers (o200k_base at 200K vocab, Llama 3 at 128K) narrow Hindi fertility to roughly 1.8, but the gap remains structural because BPE merges favor the English-dominant training corpus.

Your answer

The number 2.5 tells you that Hindi text, on average, produces 2.5 tokens per word when run through cl100k_base. Is that good? No. English on the same tokenizer scores around 1.3, so Hindi costs roughly twice as much per word across every dimension that scales with token count. **Fertility** is the metric that captures this. Tokenize a representative corpus, divide total tokens by word count, and the result tells you how efficiently the tokenizer compresses that language. Lower is always better. The reason Hindi lands at 2.5 is structural. BPE merges accumulate in proportion to training corpus frequency, and most corpora are English dominant. English words earn dedicated merged tokens; Hindi words rely more on byte level fragments and shorter merges. This imbalance is the so-called tokenizer tax: speakers of underrepresented languages pay more per character. The good news is that newer tokenizers have narrowed the gap. o200k_base (GPT-5 through GPT-5.5) cut Hindi fertility to roughly 1.8 by doubling the vocabulary to 200K. The Llama 3 tokenizer at 128K achieved similar gains. The gap is closing but has not disappeared, and 2.5 on cl100k_base is a concrete reminder of why vocabulary expansion matters.

**Fertility** measures a tokenizer's compression efficiency on a given corpus: the average number of tokens emitted per word (or per 100 characters). A fertility of 2.5 for Hindi on cl100k_base means Hindi averages 2.5 tokens per word. That is mediocre. The computation is straightforward. Tokenize a representative Hindi corpus with cl100k_base, count total tokens, divide by word count. English on the same tokenizer scores around 1.3, German around 2.1, Russian around 2.5, Tamil around 4.5. Fertility connects to three operational concerns simultaneously. **Cost**: API pricing is per token, so fertility 2.5 costs roughly twice as much as fertility 1.3 on the same logical content. **Capacity**: context windows are in tokens, so a 1M token window holds less Hindi content per word than English content. **Latency**: generation throughput is tokens per second, so the time to produce meaningful Hindi content is proportionally longer. The structural cause is the English dominance of most pretraining corpora. BPE merges accumulate where frequency is highest, so English claims the lion's share of the merge budget. Non-English text relies on shorter or byte level fragments. Frontier teams have responded by expanding vocabularies. o200k_base (200K vocab) reduced non-English fertility 20 to 30% over cl100k_base. Llama 3's 128K vocab quadrupled Llama 2's 32K. The gap narrows but does not close.

Quick reference

TL;DR

Fertility of 2.5 means Hindi averages 2.5 tokens per word on cl100k_base. English scores around 1.3, so Hindi costs roughly 2x more per word across billing, context capacity, and latency.

Memory aid

Easy to grasp

Imagine the tokenizer is a translator paid per output word. An efficient translator uses one short symbol per common input word and finishes fast. A clumsy translator needs to spell things out letter by letter, taking more symbols and costing more money. Fertility measures how many output symbols the translator needs per input word. English on most tokenizers scores around 1.3, like a fluent translator. Hindi on cl100k_base scores 2.5, like a translator who knows only basic shortcuts. Tamil can hit 4 to 5 on the same tokenizer, which is why API calls in those languages cost more per character. Newer tokenizers with bigger vocabularies have improved these numbers, but the gap is still there.

Key concepts

Fertility Tokens per Word Tokenizer Tax Multilingual Cost Efficiency

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: define fertility as tokens per word on a specific corpus, explain why 2.5 for Hindi on cl100k_base is mediocre by citing English at 1.3, connect to cost and capacity and latency, identify the structural cause in English dominated corpora, and note how o200k_base and Llama 3 have narrowed the gap.

Real-world examples

Real products, models, and research that use this idea.

Ahia et al. 2023 published a 24-language fertility benchmark on cl100k_base showing English at 1.3 tokens per word and Telugu at around 7, a 5.5x cost gap between the two languages.
OpenAI's jump from cl100k_base (100K vocab) to o200k_base (200K vocab) for GPT-4o and later models was explicitly motivated by improved multilingual and code fertility.
Meta's Llama 3 tokenizer at 128K vocab measurably reduced fertility on most non-English languages compared to Llama 2's 32K SentencePiece tokenizer.
Tiktokenizer, an open source web tool, lets you paste arbitrary text and see token counts for any tiktoken encoding, making it trivial to measure fertility on your own corpus before committing to a model.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does English fertility differ between cl100k_base and o200k_base?

QIf fertility matters so much, why not make vocabularies very large?

QHow does fertility interact with high volume multilingual deployment?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing fertility with vocabulary size. A larger vocabulary tends to reduce fertility, but they measure different things: fertility is a realized compression ratio on a corpus, vocabulary size is a count of available tokens.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.