Approximately what fraction of BPE vocabulary tokens does LiteToken identify as 'merge residues'?

BPE, short for Byte Pair Encoding, builds a vocabulary by starting from single bytes and repeatedly merging the most frequent adjacent pair into a new token. Each merge creates a brand-new vocabulary entry. The catch is that some of those entries are only useful as a stepping stone toward a bigger merge. Think of a token like `inter` that gets created at merge step 500, but by merge step 900 the tokenizer learned `international` as a single token. In most real text the longer token wins, so `inter` rarely gets emitted on its own. It still occupies a row in the embedding matrix and an output slot, but it earns its keep almost never. LiteToken, a February 2026 analysis, measured this across production tokenizers and found roughly **10%** of vocabulary slots behave like this. They named them merge residues. The takeaway: BPE vocabularies are not as tightly packed as you might assume.

About **10%**. LiteToken's headline finding is that roughly one in ten BPE vocabulary tokens is a merge residue. That is an intermediate token produced during training that almost never appears in the final tokenization of real text. The mechanism is that BPE merges are greedy and sequential. A pair merged early can later be fully absorbed into a longer token. The early token still lives in the vocabulary, but at inference the longer merge always wins, so the intermediate is a dead end. These residues waste embedding rows and `lm_head` output logits without contributing useful coverage. The practical angle is parameter budget. For a 100k vocabulary with `d_model` around 4096, 10% residues is roughly 10k unused rows times two matrices, tens of millions of parameters doing nothing. LiteToken proposes pruning residues and reallocating that capacity to tokens that actually improve bytes per token. The other distractors are wrong: under 1% drastically understates the waste, 50% is far too high, and the 256-byte base vocabulary is a tiny fixed floor, not the residue source.