Approximately what fraction of BPE vocabulary tokens does LiteToken identify as 'merge residues'?
LiteToken found that about 10% of BPE vocabulary slots are merge residues: intermediate tokens created during training that almost never appear in real tokenized text.
Imagine building a giant LEGO castle by gluing small bricks into bigger blocks, step by step. Some of those medium-sized blocks were only ever a stepping stone, because you always glued them into something larger, so the medium block never ends up in the finished castle on its own. BPE training works the same way: it merges character pairs into bigger pieces over and over. A chunk of those mid-size pieces are pure stepping stones that the final tokenizer never actually uses. LiteToken counted them and found roughly one in ten vocabulary slots is this kind of leftover, taking up space in the model without ever doing useful work.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
BPE tokenizers feel like tightly engineered artifacts: every merge was chosen because it was the most frequent adjacent pair at that step, so it is tempting to assume every slot in the final vocabulary pulls its weight. LiteToken, published in February 2026, punctures that assumption with a simple measurement. It tokenizes large corpora with production tokenizers and asks a blunt question: how often does each vocabulary token actually get emitted?
The answer surprises people. Roughly 10% of tokens almost never appear. The paper calls these merge residues. They are not corrupted, not legacy, and not the byte-level base set. They are ordinary intermediate merges that a later, longer merge always supersedes. This question matters because vocabulary slots are not free, each one buys an embedding row and an output logit, and at scale that is tens of millions of parameters.
The reason this lands as an interview topic rather than trivia is that it overturns a comfortable mental model. Most engineers picture the vocabulary as a tightly packed set of useful pieces. LiteToken shows a tenth of it is dead weight that the encoder routes around by design. We will walk through how greedy merging strands these intermediates, what they cost in parameters and compute, why they distort the way teams audit tokenizers, and how LiteToken proposes to recover the slack without retraining the whole model.
How greedy BPE strands intermediate tokens
BPE training produces an ordered list of merge rules. You start from 256 byte tokens, then repeatedly merge the most frequent adjacent pair, appending a new token each time until you hit the target vocabulary size. The order is the whole story. A merge learned early, say inter at rank 400, is available to the encoder forever after.
The problem is that later merges can fully absorb earlier ones. If the tokenizer eventually learns international as a single token at rank 1200, then any time that string appears the longer token fires. Standard BPE encoding applies merges greedily, preferring the rule that produces the longest reachable piece. So inter becomes structurally unreachable wherever the longer merge applies.
This is the key insight: a residue is not rare by chance, it is rare by construction. The encode algorithm itself routes around it. The token still sits in the vocabulary because pruning was never part of classic BPE, the algorithm only ever adds.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LiteToken (Feb 2026) audited GPT-family and Llama tokenizers and reported roughly 10% of vocabulary tokens are merge residues.
- Llama 4 Maverick ships a 128k-class vocabulary, where a 10% residue rate would imply over 12k slots worth pruning and reallocating.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you empirically identify which specific tokens are residues in a given tokenizer?
Tokenize a large representative corpus, count per-token emission frequency, and flag tokens at or near zero usage that are reachable only as merge intermediates.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming every vocabulary slot is actively used because BPE is greedy: in practice roughly 10% are dead-end merge intermediates that never surface in real text.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.