What happens to emoji ZWJ sequences under cl100k_base vs. o200k_base pre-tokenization?

**Pre-tokenization** runs before BPE and slices text into chunks using a regular expression. Wherever it makes a cut, BPE can never merge across that point. So the regex quietly decides whether an emoji ends up as one token or several. Many emoji are not a single character. A flag or a couple holding hands is built from several Unicode codepoints joined by an invisible character called the zero-width joiner, written U+200D. To a human it looks like one picture; under the hood it is a small sequence. `cl100k_base`, used by GPT-4, has a regex that often cuts at the joiner. That splits the emoji into its parts, and because BPE cannot merge across the cut, the emoji costs several tokens. `o200k_base`, used by GPT-4o, updated the regex so the whole joined sequence stays in one chunk. Now BPE can collapse a common emoji into a single token. The result is fewer tokens for emoji-heavy and multilingual text, which is exactly what GPT-4o was tuned for.

**Pre-tokenization** is the regex stage that splits raw text into chunks before BPE merges anything, and its cuts are hard boundaries BPE can never cross. That makes it the stage that decides how a ZWJ emoji tokenizes. Many emoji are multi-codepoint grapheme clusters. A rainbow flag is a base flag, a variation selector, a zero-width joiner at U+200D, and a rainbow, all rendered as one glyph. The question is whether the regex keeps that cluster together or cuts it apart. The cl100k_base regex tends to break at the joiner. The cluster fragments into separate chunks, and BPE cannot merge across the cuts, so it emits several tokens for one visible emoji. The o200k_base regex was updated to treat the whole ZWJ sequence as a single pre-token unit, so BPE can merge a frequent emoji into one token. The payoff is lower fertility for emoji-heavy and multilingual text: the same flag costs fewer tokens under o200k. This was deliberate. GPT-4o was positioned as a strongly multilingual and multimodal model, and cutting the per-emoji and per-grapheme token tax directly serves non-English users and lowers their cost.