Define STRR (Single Token Retention Rate) as a tokenizer fairness metric. Construct a concrete scenario where a language has acceptable average fertility but STRR reveals poor coverage. Explain why both metrics together are necessary for a complete picture of tokenizer fairness.
Free · 2 AI evals / day
**STRR**, the single-token retention rate, is the fraction of a language's words that the tokenizer keeps as one token rather than splitting. If 30 out of every 100 distinct words are single tokens, the STRR is 0.3.
Fertility is different. It is the average number of tokens per word measured over a real text sample. Because common words appear far more often than rare ones, fertility is dominated by the most frequent words.
Here is the scenario that shows why this matters. Suppose a language has 50 very common words, each a single token, and 5,000 less-common words that each split into four or five tokens. If common words make up 80 percent of a test corpus, the average fertility comes out around 1.7, which looks fine. But STRR over the full 5,050-word vocabulary is about 1 percent, because almost no words got single-token coverage.
So a user who writes about ordinary topics sees a low cost, while a user writing about anything specific pays the full fragmentation penalty. Fertility tells you the typical cost; STRR tells you how broadly the tokenizer covers the language. You need both to judge fairness honestly.
**STRR**, the single-token retention rate, is the fraction of a language's vocabulary that maps to exactly one token. The numerator counts words tokenized as one token; the denominator is the total distinct words.
The difference from fertility is weighting. Fertility is a corpus average, so it is frequency-weighted and the most common words dominate. STRR is computed over the distinct word list, so every word counts once.
Consider a language with 50 very common single-token words and 5,000 rare words that each fragment into 4.5 tokens. If common words are 80 percent of a corpus, fertility is about 0.8 times 1 plus 0.2 times 4.5, roughly 1.7, which reads as acceptable. STRR, though, is 50 over 5,050, about 1 percent. The average looked fine while almost the entire vocabulary fragments.
The reason both are necessary is that they answer different questions. Fertility answers what typical text costs; STRR answers how broadly the tokenizer covers the language. A few common words can hold the average down while the meaningful tail, named entities and technical terms, shatters. Reporting fertility and STRR together exposes both the average cost and the coverage gap, which is the honest fairness picture.
Quick reference
TL;DR
STRR is the fraction of a language's vocabulary that maps to single tokens; because it is unweighted by frequency, it exposes the fragmented rare-word tail that a corpus-averaged fertility number hides.
Imagine rating a buffet on how many trips it takes to get a full plate. Fertility times an average diner, who mostly grabs the popular dishes parked right by the entrance, so it looks like one easy trip. But most of the menu sits on a far table that takes four trips to reach. STRR counts how many of all the dishes are within one easy reach, not just the popular few. A buffet can feel quick to regulars while burying most of the menu. You need the regular's stopwatch and the full-menu count to know who the buffet really serves well.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Reporting STRR as the sole fairness metric without fertility alongside it STRR ignores how often words occur, so it can look poor even when the common text users actually write is cheap, a cost only fertility captures.
Comparing STRR across languages or tokenizers with different word-segmentation rules STRR is defined over a word list, so changing the boundary convention changes the denominator and makes two retention rates non-comparable.
Tuning a tokenizer to drive STRR toward 1 for a target language Forcing every rare word to a single token bloats the vocabulary and the embedding matrix, raising memory and lm_head cost for diminishing coverage value.
Quoting STRR for a deployed model as a number you can quickly improve Tokenizer choice is sticky because embeddings are indexed by token ID, so a low STRR needs continued pretraining or a retrain, not a config change.
Quick scan · 2–4 min
Everything important, quickly.
4 min: STRR is single-token coverage rate + fertility is frequency-weighted mean + worked 1.7-vs-1% gap + tail carries meaning + both needed + over-maximizing STRR bloats vocab + choice is sticky.
Real-world examples
Real products, models, and research that use this idea.
1Tokenizer fairness audits of OpenAI's o200k_base report fertility and a coverage rate together per language to check whether multilingual gains reach the rare-word tail.
2The BLOOM and Aya multilingual projects evaluated coverage breadth alongside average fertility when designing balanced tokenizers for low-resource languages.
3Cohere's enterprise localization work pairs an average-cost number with a coverage number when choosing or extending a tokenizer for a target market.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you compute STRR reproducibly for a language without a clean canonical word list?
Sign in or go Pro to see the approach for this follow-up.
QTwo candidate tokenizers tie on fertility but differ sharply on STRR. How do you choose for a legal-domain deployment?
Sign in or go Pro to see the approach for this follow-up.
QCould a high STRR itself be a problem? When?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Computing STRR over a text sample rather than the distinct word list, which sneaks the frequency bias back in and makes it just a restatement of fertility.