What constraint does BoundlessBPE relax, and what improvement does it yield?

Standard byte pair encoding, BPE, builds a vocabulary by repeatedly merging the most frequent adjacent pair of pieces. Before any of that runs, the text is chopped at spaces and punctuation by a pre-tokenizer. That chopping creates an invisible wall: BPE can merge pieces inside one word, but never across the gap between two words. **BoundlessBPE** removes that wall. Now a frequent pair that happens to straddle a space, like the two words in 'New York', can merge into a single token. The payoff is compression. The paper reports roughly a 15% gain in bytes per token, meaning each token carries more text on average. More text per token means more content fits in a fixed context window and fewer tokens are billed per request. The cost is that some tokens now cover two words at once, which can confuse tasks that expect one token to map to one word.

BoundlessBPE relaxes the **pre-tokenization boundary constraint**. Classic BPE splits text at whitespace first, then learns merges only within each word, so no token can ever span a space. BoundlessBPE drops that wall and lets the greedy merge rule operate across word boundaries too. The merge objective itself is unchanged: still pick the most frequent adjacent pair, still merge greedily. The only difference is the candidate set now includes cross-word pairs, so frequent collocations like 'in the' or 'New York' get promoted to single superword tokens. The headline result is roughly: $$\text{bytes per token} \;\approx\; +15\%$$ plus a few points of Renyi efficiency from a more uniform token distribution. More bytes per token means more text per context window and a lower token bill. The trade-off is interpretability and token to word alignment. A token covering two words breaks the one token one word assumption that NER, word alignment, and attention probing rely on. You buy compression at the cost of messier word-level tooling.