Merge residues are intermediate BPE tokens that almost never survive into final tokenizations; LiteToken found about 10% of vocab is dead weight wasting embedding and head capacity.
Imagine building a Lego castle by first snapping two bricks into a small wall, then snapping that wall into a tower. Once the tower exists, the little two-brick wall never shows up on its own again, yet it still takes a numbered slot in your instruction booklet. BPE works the same way: it builds big tokens out of smaller ones, and some of the smaller in-between pieces almost never appear by themselves in real text. LiteToken counted them and found roughly one in ten vocabulary slots is one of these unused leftovers, taking up space for no benefit.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Vocabulary efficiency is usually framed as a single dial: bigger vocabularies mean shorter sequences but heavier embedding and output layers. LiteToken's Feb 2026 finding complicates that picture by showing a portion of the vocabulary is simply dead weight, paying full parameter cost while contributing nothing to compression.
The culprit is the way BPE constructs tokens. Because merges are built in an ordered chain, the algorithm must mint intermediate tokens that later merges consume. Greedy encoding then routes around many of those intermediates, leaving them as residues. We will trace how the chain creates them, why the encoder bypasses them, what they cost on real hardware, and how pruning turns the finding into an efficiency win, with one important caveat about the long tail.
How BPE builds an ordered merge chain
BPE starts from a base alphabet, typically the 256 byte values, and repeatedly merges the most frequent adjacent pair into a new token. Crucially, the merges are ordered and dependent. If the algorithm learns that t plus h becomes th, then later it can learn th plus e becomes the. The token th had to exist before the could be defined.
This dependency is what mints intermediates. Every long, useful token like the sits at the top of a small tree of shorter merges beneath it. Some of those shorter merges are useful on their own, and some exist only to support the merge above them.
At the end of training, the vocabulary is the union of every merge ever made, base symbols included. Nothing prunes the tree; every node, useful or not, gets a permanent ID.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LiteToken (Feb 2026) audited major byte-level BPE vocabularies like the GPT-family encodings and reported roughly 10% of entries as bypassed merge residues.
- BoundlessBPE (COLM 2025) attacks the related inefficiency by allowing cross word boundary merges, lifting bytes per token by about 15%.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf you prune residues, how do you handle text that would have used one of the removed tokens?
Remap to the constituent sub-tokens that still exist, so encoding stays well-defined; accept a tiny bytes per token cost on those rare cases.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming every token in the vocabulary appears in real tokenized output, when a chunk of intermediate merge products almost never surface independently.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.