Pre-tokenization splits text before BPE merges. Why is that boundary important?

Flashcard·Easy·4.0 · 0·~30s·Asked atSap Sourcegraph Typeface·Relevant atOpenAI

Attempt it

Pre-tokenization is the regex- or rule-based step that splits raw text into coarse chunks (typically word-like fragments) before the BPE merge algorithm runs. It enforces hard boundaries: merges can never cross them. cl100k_base and o200k_base use specific regex patterns that treat whitespace, punctuation, and number-letter transitions as splits. Pre-tokenization shapes what the learned vocabulary can possibly contain.

Your answer

**Pre-tokenization** is the regex (text pattern) or rule-based step that splits raw text into coarse chunks before the BPE (byte pair encoding) merge algorithm runs. The chunks are typically word-like fragments separated by whitespace or punctuation. The key property is that pre-tokenization enforces hard boundaries. BPE merges can combine adjacent symbols within a chunk, but they can never cross from one chunk to the next. If pre-tokenization splits 'hello, world' into ['hello', ',', ' world'], then no merge will ever produce 'hello,' or ',world' as a single token. That holds no matter how frequent that pair is in the training data. The regex pattern is specific to each tokenizer. cl100k_base (GPT-4, GPT-3.5-turbo) uses one regex. o200k_base (GPT-4o, o1, o3, GPT-5) uses a different, more sophisticated regex that treats apostrophes differently. Both are published in the tiktoken source. In practice, pre-tokenization shapes what the learned vocabulary can possibly contain. A regex that splits aggressively (lots of small chunks) limits the possible merged tokens to small structures. A regex that splits loosely allows the merge algorithm to learn larger structures. The choice influences token counts, code handling efficiency, and how the tokenizer behaves on edge cases like compound words and numbers.

**Pre-tokenization** is the second stage of the tokenizer pipeline, after normalization and before the BPE merge algorithm. It uses a regex or rule set to split the input string into coarse chunks (usually word-like fragments). These chunks become the units the merge algorithm operates on. The key property is boundary enforcement. The output of pre-tokenization is a list of strings; the BPE merge algorithm runs separately within each. Merges combine symbols only within a chunk; they never cross. This is true at training (the merge algorithm only sees intra-chunk pair frequencies) and at inference (the same chunking constraint applies). The regex patterns matter because they shape the vocabulary. cl100k_base (GPT-4, GPT-3.5-turbo) splits on whitespace, punctuation, and category transitions, with special handling for English contractions like 's, 't, 're. o200k_base (GPT-4o and GPT-5) uses a more elaborate regex with better apostrophe handling and Unicode categories. Both are published in the tiktoken source. In practice, pre-tokenization determines what the tokenizer can possibly compress. A regex that splits aggressively cannot learn merged tokens spanning those boundaries. A regex that splits loosely allows larger structures but risks chaotic merges. Small changes to this regex have measurable effects on fertility and on numbers, URLs, and code. Pre-tokenization is part of the tokenizer's contract. The regex must be applied identically at training and inference. Mismatched pre-tokenization is a silent bug class similar to mismatched normalization.

TL;DR

Pre-tokenization is the regex step that splits raw text into coarse chunks before BPE merging. It enforces hard boundaries: merges can never cross them, which shapes what the vocabulary can contain.

Memory aid

Easy to grasp

Imagine BPE is a robot that walks through your text and glues adjacent pieces together into bigger units. Without any rules, the robot might glue 'hello' and 'world' together into a single weird unit because the space between them is just another character. Pre-tokenization is the step that draws walls in the text first: here is one word-ish chunk, here is the next, here is a punctuation mark on its own. The merge robot can glue pieces together inside a chunk but never across a wall. The walls come from a specific regular expression that the tokenizer's designers picked carefully. The choice of walls limits what kinds of merged tokens can ever exist in the vocabulary.

Key concepts

Pre-tokenization Regex Pattern Boundary

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Pre-tokenization is the step in the tokenizer pipeline that most users never look at directly but that quietly shapes everything downstream. It is the regex or rule set that splits the input string into coarse chunks before the BPE merge algorithm runs. The choice of regex determines what merges are even possible to learn, which in turn determines how the tokenizer behaves on real-world text.

Pre-tokenization in the pipeline

Tokenizer pipelines have four sequential stages. Normalization runs first (a string transformation for Unicode and case handling). Pre-tokenization runs second (splitting the string into a list of chunks). The model runs third (BPE merges, or SentencePiece-Unigram segmentation, operating on each chunk independently). Post-processing runs last (insertion of special tokens like BOS, EOS, role markers).

Pre-tokenization is the bridge between text and the merge algorithm. Its input is a string. Its output is a list of strings, each of which the merge algorithm will tokenize separately. The chunks are usually word-like: a leading whitespace plus a sequence of letters, or a punctuation character, or a digit sequence, or a piece of whitespace.

The Hugging Face tokenizers library makes this stage explicit. A tokenizer.json file has a pre_tokenizer field that describes the splitter as a structured config: which regex to use, how to handle whitespace, whether to preserve byte-level encoding. The tiktoken library does the same thing in code: each encoding (cl100k_base, o200k_base) has a published regex that runs as the pre-tokenization step.

The key consequence of this pipeline shape is that the merge algorithm never sees a single coherent input string. It sees a list of chunks. Adjacent characters in the original input that fall into different chunks are not adjacent from the merge algorithm's perspective. They are in different lists and cannot be combined.

The hard boundary constraint

The cl100k_base regex, in detail

What o200k_base changed

Downstream consequences of the regex

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI's tiktoken source publishes the exact regex for cl100k_base and o200k_base. You can inspect them in the encoding registry and even compile and apply them yourself in Python.
The cl100k_base regex specifically handles English contractions ('s, 't, 're, 've, 'm, 'll, 'd) as their own chunks, which is why 'don't' tokenizes consistently across casing variations.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy do cl100k_base and o200k_base use different regex patterns?

o200k_base shipped with GPT-4o in 2024, with a vocabulary about twice as large and much better multilingual compression. The regex changes are: contractions attach to the preceding word instead of forming their own chunks; letter classes are case-aware and include combining marks, so an uppercase run followed by lowercase letters is split; and punctuation runs can absorb a trailing '/'. Digit grouping is the same in both encodings (runs of one to three digits). Compare token counts under both encodings on your own text to see the effect.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing pre-tokenization with tokenization. Pre-tokenization is the splitting step that runs before BPE merges; the merges then operate within each pre-tokenization chunk separately.

60 second bullets to scan on the way to the call.

Define pre-tokenization in one sentence.
Identify where it sits in the tokenizer pipeline (after normalization, before merging).

Primary sources. Browse if you want the original framing.