**Pre-tokenization** is the very first step a tokenizer like `cl100k_base` runs, before any Byte Pair Encoding happens. BPE is the algorithm that builds tokens by repeatedly gluing the most frequent neighbouring pieces together. Pre-tokenization decides where BPE is even allowed to glue.
It works by applying a regular expression, a text pattern, to the raw input string. That pattern slices the text into coarse chunks: a run of letters, a run of digits, a punctuation mark, a stretch of whitespace. Each slice is handed to BPE separately.
The important consequence is that those slice edges are hard boundaries. BPE counts frequencies and merges pieces only inside one chunk, never across two. So a single word can become one token, but a token can never stretch across a space into the next word in `cl100k_base`. The regex, not BPE, is what enforces that rule.
The wrong answers describe other jobs, such as compressing text, normalizing Unicode, or tagging entities, that pre-tokenization does not do.
The primary purpose of regex-based **pre-tokenization** is to split the raw input into coarse chunks before BPE merging runs. The constraint it imposes is that BPE can never merge across a chunk boundary.
Mechanically, `cl100k_base` applies a regex that breaks text at the transitions that matter: whitespace, punctuation, and letter vs digit boundaries. It also keeps a leading space attached to the word that follows. Each resulting chunk is then fed to BPE independently.
The boundary is absolute. BPE's frequency counting and greedy merging operate strictly within a chunk. So it can learn a single token for a whole word, but never one that spans 'word + space + word'. That is the key structural prior: the regex designer is deciding up front which boundaries are linguistically meaningful, and BPE only optimizes inside those lines.
This is also why the regex is a real design lever. `o200k_base` shipped an updated pattern, mainly to handle Unicode ZWJ emoji sequences and multilingual character runs more gracefully. Same BPE algorithm, different pre-tokenization rules, and the resulting token vocabularies differ in exactly the places the regex treats differently.
Quick reference
TL;DR
Pre-tokenization splits raw text into coarse chunks first, and those chunk edges become hard walls that BPE merges can never cross.
Imagine cutting a long ribbon into shorter pieces before you start gluing beads onto each piece. Once the ribbon is cut, you can only glue beads that sit on the same piece, never across a cut. Pre-tokenization is that cutting step: a pattern slices the text at spaces, punctuation, and digit-letter changes. Then the merging step that glues frequent neighbours together only works inside each slice. So a token like a word can form, but a token that spans 'word + space + next word' cannot, because the cut sits between them.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Training a tokenizer for a language with no whitespace, like Japanese or Chinese Whitespace-based pre-tokenization gives almost no useful boundaries; a raw-text approach like SentencePiece fits better.
You actually want multi-word units as single tokens for a phrase-heavy domain Regex pre-tokenization forbids cross-word merges by design, so it blocks the very tokens you want.
Comparing token counts across two different encodings Each encoding ships its own pre-tokenization regex, so the chunking and the resulting counts are not interchangeable.
Quick scan · 2–4 min
Everything important, quickly.
3 min: regex splits raw text + chunk edges are hard walls + BPE merges only inside a chunk + no cross-word tokens + leading-space rule + cl100k vs o200k regex differences + pin encoding per model.
Code example
python
import regex as re
# A simplified cl100k_base-style pre-tokenization pattern.
# The real pattern also handles contractions, leading spaces, digits.
pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+""")
text = "Hello world 42!"
chunks = pat.findall(text)
print(chunks) # ['Hello', ' world', ' 42', '!']
# BPE merges run INSIDE each chunk only.
# It can build a token for ' world', but never for 'Hello world'
# because the two sit in different chunks.
Real-world examples
Real products, models, and research that use this idea.
1OpenAI's tiktoken ships the cl100k_base and o200k_base regex patterns directly in its source, and o200k_base powers the GPT-5.5 family.
2HuggingFace tokenizers exposes pluggable pre-tokenizers (ByteLevel, Whitespace, Split), letting you compose the same boundary logic for custom BPE vocabularies.
3Llama 4's HuggingFace tokenizer.json encodes its own pre-tokenization rules, which is why its token boundaries differ from OpenAI encodings on the same text.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat would the vocabulary look like if you dropped pre-tokenization entirely and let BPE merge across spaces?
Sign in or go Pro to see the approach for this follow-up.
QHow do cl100k_base and o200k_base regex patterns differ, and what user populations does that affect?
Sign in or go Pro to see the approach for this follow-up.
QWhy can't you mix a cl100k token count with an o200k model when budgeting cost?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Thinking pre-tokenization cleans or normalizes the text by stripping punctuation or lowercasing, when its real job is to draw merge boundaries BPE cannot cross.