Pre-tokenization is the regex- or rule-based step that splits raw text into coarse chunks (typically word-like fragments) before the BPE merge algorithm runs. It enforces hard boundaries: merges can never cross them. cl100k_base and o200k_base use specific regex patterns that treat whitespace, punctuation, and number-letter transitions as splits. Pre-tokenization shapes what the learned vocabulary can possibly contain.
**Pre-tokenization** is the regex or rule-based step that splits raw text into coarse chunks before the BPE merge algorithm runs. The chunks are typically word-like fragments separated by whitespace or punctuation.
The key property is that pre-tokenization enforces hard boundaries. BPE merges can combine adjacent symbols within a chunk, but they can never cross from one chunk to the next. If pre-tokenization splits 'hello, world' into ['hello', ',', ' world'], then no merge will ever produce 'hello,' or ',world' as a single token, no matter how frequent that pair is in the training data.
The regex pattern is specific to each tokenizer. cl100k_base (GPT-4, GPT-3.5-turbo) uses one regex; o200k_base (GPT-4o, o1, o3, GPT-5, GPT-5.5) uses a different, more sophisticated regex that treats numbers and apostrophes differently. Both are published in the tiktoken source.
Why this matters. Pre-tokenization shapes what the learned vocabulary can possibly contain. A regex that splits aggressively (lots of small chunks) limits the possible merged tokens to small structures; a regex that splits loosely allows the merge algorithm to learn larger structures. The choice influences fertility, code-handling efficiency, and how the tokenizer behaves on edge cases like compound words and numbers.
**Pre-tokenization** is the second stage of the modern tokenizer pipeline, after normalization and before the BPE merge algorithm. It uses a regex or rule set to split the input string into coarse chunks (usually word-like fragments) that become the units the merge algorithm operates on.
The load-bearing property is boundary enforcement. The output of pre-tokenization is a list of strings; the BPE merge algorithm runs separately within each. Merges combine symbols only within a chunk; they never cross. This is true at training (the merge algorithm only sees intra-chunk pair frequencies) and at inference (the same chunking constraint applies).
The regex patterns matter because they shape the vocabulary. cl100k_base (GPT-4, GPT-3.5-turbo) splits on whitespace, punctuation, and category transitions, with special handling for English contractions like 's, 't, 're. o200k_base (GPT-4o through GPT-5.5) uses a more elaborate regex with better digit grouping, apostrophe handling, and Unicode categories. Both are published in the tiktoken source.
Why this matters in practice. Pre-tokenization determines what the tokenizer can possibly compress. A regex that splits aggressively cannot learn merged tokens spanning those boundaries; a regex that splits loosely allows larger structures but risks chaotic merges. Frontier teams spend significant care on this regex; small changes have measurable effects on fertility and on numbers, URLs, and code.
Pre-tokenization is part of the tokenizer's contract. The regex must be applied identically at training and inference; mismatched pre-tokenization is a silent bug class similar to mismatched normalization.
Quick reference
TL;DR
Pre-tokenization is the regex step that splits raw text into coarse chunks before BPE merging. It enforces hard boundaries: merges can never cross them, which shapes what the vocabulary can contain.
Imagine BPE is a robot that walks through your text and glues adjacent pieces together into bigger units. Without any rules, the robot might glue 'hello' and 'world' together into a single weird unit because the space between them is just another character. Pre-tokenization is the step that draws walls in the text first: here is one word-ish chunk, here is the next, here is a punctuation mark on its own. The merge robot can glue pieces together inside a chunk but never across a wall. The walls are drawn by a specific regular expression that the tokenizer's designers picked very carefully, because the choice of walls limits what kinds of merged tokens can ever exist in the vocabulary.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Trying to make BPE merges cross word boundaries to capture multi-word expressions. The pre-tokenization regex enforces hard boundaries on whitespace; no amount of training will produce a token spanning two words because the merge algorithm never sees those pairs.
Reusing a tokenizer's regex pattern across model families without verification. Each tokenizer has its own regex; cl100k_base and the Llama 3 tokenizer use different patterns. The same input can chunk differently under different regexes.
Treating pre-tokenization as unimportant when training a custom tokenizer. The regex is one of the most consequential design choices. A poorly designed pre-tokenization regex produces a tokenizer with high fertility, poor code handling, or odd multilingual behavior that is hard to fix without re-training.
Quick scan · 2–4 min
Everything important, quickly.
5 min: define pre-tokenization, place it in the pipeline between normalization and merging, explain the hard-boundary constraint on the merge algorithm, describe the regex pattern at a high level, contrast cl100k_base and o200k_base regexes briefly, and connect to the leading-space token effect.
Real-world examples
Real products, models, and research that use this idea.
1OpenAI's tiktoken source publishes the exact regex for cl100k_base and o200k_base; you can inspect them in the encoding registry and even compile and apply them yourself in Python.
2The cl100k_base regex specifically handles English contractions (s, t, re, ve, m, ll, d) as their own chunks, which is why 'don't' tokenizes consistently across casing variations.
3o200k_base introduced number grouping by threes, improving numeric handling for the o-series reasoning models like o1 and o3 that needed better arithmetic encoding.
4Hugging Face's tokenizers library exposes pre-built pre-tokenizers (Whitespace, Punctuation, Digits, ByteLevel, Sequence) that can be composed; the ByteLevel pre-tokenizer is what tiktoken-style tokenizers use under the hood.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy do cl100k_base and o200k_base use different regex patterns?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the relationship between the pre-tokenization regex and the leading-space token effect?
Sign in or go Pro to see the approach for this follow-up.
QIf pre-tokenization is so important, why is it not as widely discussed as BPE merges?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Confusing pre-tokenization with tokenization. Pre-tokenization is the splitting step that runs before BPE merges; the merges then operate within each pre-tokenization chunk separately.