Explain the role of regex based pre-tokenization in cl100k_base (GPT-4) and o200k_base (GPT-4o). Focus specifically on how each handles Unicode ZWJ (Zero Width Joiner) emoji sequences, and why the o200k_base regex was updated.
cl100k_base splits ZWJ emoji at the joiner so they fragment into many tokens; o200k_base keeps the whole sequence in one chunk, so common emoji tokenize far more cheaply.
Picture a sticker made of three smaller stickers taped together with invisible tape: a face, a tape strip, and a heart that together mean one thing. The older system sees the invisible tape and rips the sticker into separate pieces, charging you for each one. The newer system recognizes the taped-together sticker as a single unit and can give it one price tag. The invisible tape is the zero-width joiner, an unprintable character that glues emoji parts into one symbol, and the price tag is how many tokens the text costs.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Emoji look like single characters, but to a tokenizer they are often small programs: a base symbol, an invisible joiner, and one or more follow-on symbols that a renderer fuses into one glyph. That gap between what you see and what the bytes contain is exactly where tokenization gets interesting, and it is why the same rainbow flag can cost six tokens on one OpenAI model and one or two on another.
This is a senior-level question because it forces you to connect three layers that are usually discussed separately: Unicode grapheme clusters, the pre-tokenization regex, and the BPE merge constraint. Get any one wrong and the explanation collapses. People who think emoji are single codepoints cannot explain the fragmentation; people who think BPE can stitch fragments back together miss the hard-boundary rule; people who credit everything to vocabulary size miss that the regex is the real lever here.
We will build it up in order: what a ZWJ sequence actually is in bytes, how cl100k_base shatters it, how o200k_base keeps it whole, and why OpenAI made that change deliberately as part of GPT-4o's multilingual positioning. By the end you should be able to predict the token-count difference and explain the production consequence for cost.
What a ZWJ emoji sequence is in bytes
A surprising number of emoji are not single Unicode codepoints. They are grapheme clusters: several codepoints that a text renderer combines into one visible glyph. The glue is the zero-width joiner, codepoint U+200D, which prints nothing but tells the renderer to fuse its neighbours. A rainbow flag, for example, is a waving white flag, a variation selector that requests emoji presentation, the joiner at U+200D, and a rainbow. Four codepoints, one glyph.
In UTF-8, each of those codepoints expands to multiple bytes. The joiner alone is three bytes, and the emoji codepoints are four bytes each. So a single rainbow flag is well over a dozen raw bytes before any tokenization decision is made. Skin-tone variants, family emoji, and many flags follow the same pattern: a base plus modifiers plus joiners.
This is the crux. Because the visible glyph is built from a sequence, the question of how many tokens it costs is really the question of whether the tokenizer keeps that sequence together or treats the joiner as a place to cut. Nothing about the picture you see determines that; the regex does.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import tiktoken
cl100k = tiktoken.get_encoding("cl100k_base")
o200k = tiktoken.get_encoding("o200k_base")
flag = "\U0001F3F3\uFE0F\u200D\U0001F308" # rainbow flag (ZWJ sequence)
print(len(cl100k.encode(flag))) # more tokens: cluster fragments at the ZWJ
print(len(o200k.encode(flag))) # fewer tokens: ZWJ sequence kept intact
# Always resolve the encoding from the model so the regex matches:
enc = tiktoken.encoding_for_model("gpt-4o") # -> o200k_base| Aspect | cl100k_base (GPT-4) | o200k_base (GPT-4o) |
|---|---|---|
| ZWJ emoji sequence | Split at the joiner into components | Kept as one pre-token unit |
| Tokens per ZWJ flag emoji | Several (multi-byte fallback) | One to two if frequent |
| Emoji modifiers (skin tone, flags) | Often fragmented | Recognized as a unit |
| Multilingual character runs | Awkward boundaries | Grouped more sensibly |
| Design motivation | ASCII-tuned | Multilingual and multimodal |
Real products, models, and research that use this idea.
- OpenAI's o200k_base, shipped with the GPT-5.5 family, recognizes ZWJ emoji and modifier sequences so common emoji cost fewer tokens than under cl100k_base.
- Messaging and social products with heavy emoji usage see lower per-message token bills when they move to o200k-based models for the same text.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you estimate the token-cost savings of moving an emoji-heavy chat product from a cl100k model to a GPT-4o model?
Sample real messages, encode under both encodings with tiktoken, weight the per-message token delta by traffic volume.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a ZWJ emoji is one codepoint, so it should be one token; it is actually several codepoints glued by U+200D, and the regex decides whether they stay together.
60 second bullets to scan on the way to the call.
What a ZWJ emoji sequence is at the codepoint level
Why pre-tokenization cuts are walls BPE cannot cross
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.