Standard BPE merges the most frequent adjacent pair by raw token count across the entire corpus. Describe what would change: both algorithmically and in terms of the resulting vocabulary: if instead you weighted merge priority by document frequency (number of documents containing the pair) rather than total occurrence count.
Free · 2 AI evals / day
Standard **BPE** scores each adjacent pair by raw count: every single occurrence anywhere in the corpus adds one. A pair that appears 10,000 times inside one long document contributes all 10,000 to its score.
Document frequency changes the counting rule. It asks how many distinct documents contain the pair, and caps each document's contribution at one. That 10,000-occurrence pair now scores one if it lives in a single document.
Algorithmically, only the scoring function changes; the merge loop is identical. But the rankings shift. Jargon repeated heavily in a few documents drops, while pairs that show up in nearly every document, like common function words and morphemes, rise.
The vocabulary that results generalizes better across diverse text and resists being hijacked by repetitive outlier documents. The trade is that dense technical domains, where in-document repetition is real signal, get fewer dedicated tokens and tokenize less efficiently.
Only the **scoring function** changes; the count, select, replace loop is untouched. Standard BPE scores a pair by raw occurrences, so one document can contribute thousands. Document frequency caps each document at one vote.
$$\text{score}(a,b) = \bigl|\{\, d : (a,b) \in d \,\}\bigr|$$
This is the IDF intuition from TF-IDF. It rewards patterns that are broadly distributed, not just locally dense.
The ranking effect is the substance. A pair like 'whereas party', repeated 5,000 times in one legal document, plummets from 5,000 to 1. Function words and common morphemes that appear in nearly every document rise to the top.
The resulting vocabulary generalizes across diverse documents and resists capture by high-repetition outliers. It spends its budget on broadly useful subwords rather than corpus-specific jargon.
The cost is real: domains where in-document repetition is the actual signal, code, DNA, log files, get underweighted. Their dense patterns earn fewer tokens, so the tokenizer compresses them less efficiently. It is a generalization versus specialization trade.
Quick reference
TL;DR
Weighting BPE merges by document frequency caps each document at one vote, so high-repetition outliers stop dominating and the vocabulary generalizes more broadly.
Imagine the team votes on which word pairs deserve their own shortcut. Under the usual rule, each person votes once per time they say a phrase, so one very chatty lawyer who repeats 'whereas party' five thousand times basically decides the outcome alone.
Document-frequency weighting changes the rule to one document, one vote. The lawyer's repetition now counts the same as anyone else who used the phrase at all. The shortcuts that win are the ones lots of different people use, not the ones one person hammers. The result is a set of shortcuts that serves the whole team, at the cost of fewer shortcuts tuned to that one chatty corner.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tokenizers targeting code, DNA, or log data where in-file repetition is the signal Document frequency caps each file at one vote, underweighting dense repeated patterns and fragmenting them into more tokens.
Pipelines that already deduplicate the corpus heavily before training Deduplication already removes the outlier-domination problem, so switching the scoring metric adds complexity for little gain.
Single-domain corpora with uniform document length and style With little variance in how documents use phrases, document and term frequency rank pairs almost identically, so the change buys nothing.
Quick scan · 2–4 min
Everything important, quickly.
3 min: scoring-only change + raw count vs document frequency + IDF analogy + vocabulary reallocation + dense-domain cost + dedup as the production alternative.
Real-world examples
Real products, models, and research that use this idea.
1LLM pretraining pipelines (Llama 4, GPT-5.5 era) aggressively deduplicate web corpora, which addresses the same outlier-domination problem that document-frequency weighting targets.
2Code tokenizers benefit from raw-count weighting because repeated identifiers and indentation patterns within a file are genuine signal, not noise.
3TF-IDF in classic search ranking uses the same document-frequency intuition to downweight terms that are locally dense but globally uninformative.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would document-frequency weighting interact with corpus deduplication, and would you still want it after deduping?
Sign in or go Pro to see the approach for this follow-up.
QCould you blend raw and document frequency, and what objective would the blend optimize?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming document-frequency weighting is strictly better. It generalizes across documents but underweights dense domains like code or DNA where in-document repetition is the signal.