Explain the role of regex-based pre-tokenization in cl100k_base (GPT-4) and o200k_base (GPT-4o). Focus specifically on how each handles Unicode ZWJ (Zero-Width Joiner) emoji sequences, and why the o200k_base regex was updated.
Free · 2 AI evals / day
**Pre-tokenization** runs before BPE and slices text into chunks using a regular expression. Wherever it makes a cut, BPE can never merge across that point. So the regex quietly decides whether an emoji ends up as one token or several.
Many emoji are not a single character. A flag or a couple holding hands is built from several Unicode codepoints joined by an invisible character called the zero-width joiner, written U+200D. To a human it looks like one picture; under the hood it is a small sequence.
`cl100k_base`, used by GPT-4, has a regex that often cuts at the joiner. That splits the emoji into its parts, and because BPE cannot merge across the cut, the emoji costs several tokens.
`o200k_base`, used by GPT-4o, updated the regex so the whole joined sequence stays in one chunk. Now BPE can collapse a common emoji into a single token. The result is fewer tokens for emoji-heavy and multilingual text, which is exactly what GPT-4o was tuned for.
**Pre-tokenization** is the regex stage that splits raw text into chunks before BPE merges anything, and its cuts are hard boundaries BPE can never cross. That makes it the stage that decides how a ZWJ emoji tokenizes.
Many emoji are multi-codepoint grapheme clusters. A rainbow flag is a base flag, a variation selector, a zero-width joiner at U+200D, and a rainbow, all rendered as one glyph. The question is whether the regex keeps that cluster together or cuts it apart.
The cl100k_base regex tends to break at the joiner. The cluster fragments into separate chunks, and BPE cannot merge across the cuts, so it emits several tokens for one visible emoji. The o200k_base regex was updated to treat the whole ZWJ sequence as a single pre-token unit, so BPE can merge a frequent emoji into one token.
The payoff is lower fertility for emoji-heavy and multilingual text: the same flag costs fewer tokens under o200k. This was deliberate. GPT-4o was positioned as a strongly multilingual and multimodal model, and cutting the per-emoji and per-grapheme token tax directly serves non-English users and lowers their cost.
Quick reference
TL;DR
cl100k_base splits ZWJ emoji at the joiner so they fragment into many tokens; o200k_base keeps the whole sequence in one chunk, so common emoji tokenize far more cheaply.
Picture a sticker made of three smaller stickers taped together with invisible tape: a face, a tape strip, and a heart that together mean one thing. The older system sees the invisible tape and rips the sticker into separate pieces, charging you for each one. The newer system recognizes the taped-together sticker as a single unit and can give it one price tag. The invisible tape is the zero-width joiner, an unprintable character that glues emoji parts into one symbol, and the price tag is how many tokens the text costs.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Budgeting tokens for a GPT-4 or GPT-3.5-turbo call Those models use cl100k_base, so o200k emoji-friendly counts would understate their actual token usage.
Estimating cost for rare or novel emoji not seen during training Even with the better regex, an unmerged rare emoji still falls back to component tokens, so the savings do not apply.
Comparing fertility across tokenizers from different vendors Each ships its own pre-tokenization rules for ZWJ and grapheme clusters, so the emoji handling is not directly comparable.
Quick scan · 2–4 min
Everything important, quickly.
4 min: ZWJ emoji are multi-codepoint clusters + pre-tokenization cuts are walls + cl100k splits at the joiner + o200k keeps the sequence intact + fewer tokens for emoji and CJK + driven by GPT-4o multilingual goals + counts not portable across encodings.
Code example
python
import tiktoken
cl100k = tiktoken.get_encoding("cl100k_base")
o200k = tiktoken.get_encoding("o200k_base")
flag = "\U0001F3F3\uFE0F\u200D\U0001F308" # rainbow flag (ZWJ sequence)
print(len(cl100k.encode(flag))) # more tokens: cluster fragments at the ZWJ
print(len(o200k.encode(flag))) # fewer tokens: ZWJ sequence kept intact
# Always resolve the encoding from the model so the regex matches:
enc = tiktoken.encoding_for_model("gpt-4o") # -> o200k_base
Side-by-side
Aspect
cl100k_base (GPT-4)
o200k_base (GPT-4o)
ZWJ emoji sequence
Split at the joiner into components
Kept as one pre-token unit
Tokens per ZWJ flag emoji
Several (multi-byte fallback)
One to two if frequent
Emoji modifiers (skin tone, flags)
Often fragmented
Recognized as a unit
Multilingual character runs
Awkward boundaries
Grouped more sensibly
Design motivation
ASCII-tuned
Multilingual and multimodal
Real-world examples
Real products, models, and research that use this idea.
1OpenAI's o200k_base, shipped with the GPT-5.5 family, recognizes ZWJ emoji and modifier sequences so common emoji cost fewer tokens than under cl100k_base.
2Messaging and social products with heavy emoji usage see lower per-message token bills when they move to o200k-based models for the same text.
3tiktoken exposes both encodings, so a team can encode the same emoji string under each and watch the token count drop on o200k_base.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you estimate the token-cost savings of moving an emoji-heavy chat product from a cl100k model to a GPT-4o model?
Sign in or go Pro to see the approach for this follow-up.
QWhy doesn't keeping ZWJ sequences together help if the specific emoji never appeared in training?
Sign in or go Pro to see the approach for this follow-up.
QBeyond emoji, what other Unicode constructs benefit from the o200k pre-tokenization change?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming a ZWJ emoji is one codepoint, so it should be one token; it is actually several codepoints glued by U+200D, and the regex decides whether they stay together.