LiteToken found that about 5% to 10% of BPE vocabulary slots are merge residues: intermediate tokens created during training that almost never appear in real tokenized text.
Imagine building a giant LEGO castle by gluing small bricks into bigger blocks, step by step. Some of those medium-sized blocks were only ever a stepping stone, because you always glued them into something larger. So the medium block never ends up in the finished castle on its own. BPE training works the same way: it merges character pairs into bigger pieces over and over. A chunk of those mid-size pieces are pure stepping stones that the final tokenizer never uses. LiteToken counted them and found roughly one in ten to one in twenty vocabulary slots is this kind of leftover. It takes up space in the model without ever doing useful work.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
BPE tokenizers feel like tightly engineered artifacts. Every merge was chosen because it was the most frequent adjacent pair at that step. So it is tempting to assume every slot in the final vocabulary pulls its weight. LiteToken, published in February 2026, punctures that assumption with a simple measurement. It tokenizes large corpora with production tokenizers and asks a blunt question: how often does each vocabulary token get emitted?
About 5% to 10% of tokens almost never appear. The paper calls these merge residues. They are not corrupted, not legacy, and not the byte-level base set. They are ordinary intermediate merges that a later, longer merge always supersedes. This matters because vocabulary slots are not free. Each one buys an embedding row and an output logit, and at scale that is tens of millions of parameters.
The finding overturns a comfortable mental model. Most engineers picture the vocabulary as a tightly packed set of useful pieces. LiteToken shows that a twentieth to a tenth of it is dead weight that the encoder routes around by design.
How greedy BPE strands intermediate tokens
BPE training produces an ordered list of merge rules. You start from 256 byte tokens, then repeatedly merge the most frequent adjacent pair. Each merge appends a new token until you hit the target vocabulary size. The order is the whole story. A merge learned early, say inter at rank 400, is available to the encoder forever after.
The problem is that later merges can fully absorb earlier ones. If the tokenizer eventually learns international as a single token at rank 1200, then any time that string appears the longer token fires. Standard BPE encoding applies merges in rank order, so inter forms first. Later merges then consume it on the way to international. So inter rarely survives into the output wherever the longer merge applies.
A residue is not rare by chance. It is rare by construction. The encode algorithm itself routes around it. The token still sits in the vocabulary because pruning was never part of classic BPE. The algorithm only ever adds.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LiteToken (Feb 2026) audited Qwen, DeepSeek-V3, BLOOM, Gemma 3, Llama 3, and o200k_base tokenizers and flagged about 5% to 10% of vocabulary tokens as merge residues.
- Llama 4 Maverick ships a vocabulary of about 202k tokens, where a 5% to 10% residue rate would imply roughly 10k to 20k slots worth removing.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you empirically identify which specific tokens are residues in a given tokenizer?
Tokenize a large representative corpus, count per-token emission frequency, and flag tokens at or near zero usage that are reachable only as merge intermediates.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming every vocabulary slot is actively used because BPE is greedy: in practice about 5% to 10% are dead-end merge intermediates that never surface in real text.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.