The answer is **SentencePiece**. To see why, you have to look at a step most tokenizers run first, called pre-tokenization.
Standard BPE and WordPiece, the tokenizer BERT uses, split text on whitespace before they do any merging. That split is a hard wall: the merge step can only glue pieces that sit inside the same word, so it can never learn a token that crosses a space.
SentencePiece is built differently. It treats the raw text as one continuous stream and turns each space into an ordinary character, shown as the symbol ▁. Because the space is now just another character, merges are free to cross what used to be a word boundary, and SentencePiece can learn a single token like '▁New▁York'.
That design is exactly why SentencePiece suits languages with no spaces, like Japanese and Chinese. The distractors are wrong because tiktoken and byte-level BPE still pre-split with a regex, so they keep the space wall in place.
**SentencePiece** is the tokenizer that can learn merges across whitespace by design, because it never pre-splits on spaces.
Most tokenizers run a pre-tokenization step first. Standard BPE and WordPiece split text on whitespace, which makes the space a hard boundary: the merge algorithm only operates inside a word, so a cross-word token is impossible. SentencePiece skips that step entirely. It consumes raw text as a continuous stream and encodes each space as a regular character, the meta-symbol ▁. With the space demoted to an ordinary character, merges can span what used to be a word boundary, so a frequent phrase like '▁New▁York' can become a single token.
This is not just a curiosity. It is what makes SentencePiece language-agnostic. Japanese and Chinese have no whitespace between words, so a whitespace pre-splitting tokenizer has nothing useful to split on, while SentencePiece treats those scripts the same as any other character stream.
The distractors all keep the space wall. The tiktoken regex pre-tokenizer cuts at spaces. Byte-level BPE, despite operating on bytes, still applies a regex pre-tokenizer that prevents cross-boundary merges in practice. WordPiece pre-splits as well.
Quick reference
TL;DR
SentencePiece, because it runs on raw text and treats whitespace as an ordinary character (▁), so merges are free to cross word boundaries.
Imagine writing a sentence on a long strip of paper with no gaps marked as special. Most tokenizers first snip the strip at every space, so they can only join letters that sit inside one word. SentencePiece skips the snipping. It treats a space as just another mark on the paper, so it can join letters and even whole words across what used to be a gap. That freedom is exactly what you need for languages like Japanese or Chinese, which do not put spaces between words at all.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You specifically want clean, reusable intra-word subwords with no phrase fragments Cross-word merges can fill the vocabulary with brittle phrase units, so a whitespace pre-splitting tokenizer fits better.
Serving an OpenAI model whose encoding is fixed Those models use byte-level BPE with a regex pre-tokenizer, so SentencePiece behavior is not what the model expects.
A purely English, short-text workload The cross-word and space-free advantages barely apply, so the SentencePiece benefit over GPT-style BPE is marginal.
Quick scan · 2–4 min
Everything important, quickly.
3 min: standard BPE pre-splits on whitespace + that wall blocks cross-word merges + SentencePiece skips the pre-split + encodes space as ▁ + merges cross former boundaries + suits space-free scripts + tiktoken and byte-level BPE keep the wall.
Code example
python
import sentencepiece as spm
sp = spm.SentencePieceProcessor(model_file="spm.model")
# Spaces become the meta-symbol U+2581 (shown as the lower one-eighth block).
print(sp.encode("New York", out_type=str))
# e.g. ['\u2581New', '\u2581York'] -> a frequent phrase could merge to ['\u2581New\u2581York']
# No whitespace? No problem: the raw stream is tokenized directly.
print(sp.encode("\u6771\u4eac\u90fd", out_type=str)) # Tokyo (no spaces between chars)
Real-world examples
Real products, models, and research that use this idea.
1Llama 4 and T5 use SentencePiece, which is why their tokenizers handle space-free scripts without a separate word segmenter.
2Multilingual products targeting Japanese and Chinese pick SentencePiece-based models because there is no whitespace to pre-split on.
3Gemma 4 ships a SentencePiece tokenizer, letting common multi-word and CJK units land as single tokens.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat is the downside of letting merges cross word boundaries, and how do you control it?
Sign in or go Pro to see the approach for this follow-up.
QHow does SentencePiece tokenize a language like Chinese that has no spaces at all?
Sign in or go Pro to see the approach for this follow-up.
QWhy does GPT-style byte-level BPE keep a regex pre-tokenizer if it costs cross-word merges?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Thinking byte-level BPE allows cross-word merges because bytes ignore whitespace; its regex pre-tokenizer still cuts at spaces and blocks them.