What is the primary purpose of regex-based pre-tokenization in cl100k_base, and what constraint does it impose on BPE?

**Pre-tokenization** is the very first step a tokenizer like `cl100k_base` runs, before any Byte Pair Encoding happens. BPE is the algorithm that builds tokens by repeatedly gluing the most frequent neighbouring pieces together. Pre-tokenization decides where BPE is even allowed to glue. It works by applying a regular expression, a text pattern, to the raw input string. That pattern slices the text into coarse chunks: a run of letters, a run of digits, a punctuation mark, a stretch of whitespace. Each slice is handed to BPE separately. The important consequence is that those slice edges are hard boundaries. BPE counts frequencies and merges pieces only inside one chunk, never across two. So a single word can become one token, but a token can never stretch across a space into the next word in `cl100k_base`. The regex, not BPE, is what enforces that rule. The wrong answers describe other jobs, such as compressing text, normalizing Unicode, or tagging entities, that pre-tokenization does not do.

The primary purpose of regex-based **pre-tokenization** is to split the raw input into coarse chunks before BPE merging runs. The constraint it imposes is that BPE can never merge across a chunk boundary. Mechanically, `cl100k_base` applies a regex that breaks text at the transitions that matter: whitespace, punctuation, and letter vs digit boundaries. It also keeps a leading space attached to the word that follows. Each resulting chunk is then fed to BPE independently. The boundary is absolute. BPE's frequency counting and greedy merging operate strictly within a chunk. So it can learn a single token for a whole word, but never one that spans 'word + space + word'. That is the key structural prior: the regex designer is deciding up front which boundaries are linguistically meaningful, and BPE only optimizes inside those lines. This is also why the regex is a real design lever. `o200k_base` shipped an updated pattern, mainly to handle Unicode ZWJ emoji sequences and multilingual character runs more gracefully. Same BPE algorithm, different pre-tokenization rules, and the resulting token vocabularies differ in exactly the places the regex treats differently.