What happens to emoji ZWJ sequences under cl100k_base vs. o200k_base pre-tokenization?

Short answer·Hard·4.0 · 0·~3 min·Asked atGnani Kore Ai OpenAI

Attempt it

Explain the role of regex based pre-tokenization in cl100k_base (GPT-4) and o200k_base (GPT-4o). Focus specifically on how each handles Unicode ZWJ (Zero Width Joiner) emoji sequences, and why the o200k_base regex was updated.

Free · 2 AI evals / day

**Pre-tokenization** runs before BPE (byte pair encoding) and slices text into chunks using a regular expression, a text-matching pattern. Wherever it makes a cut, BPE can never merge across that point. So the regex decides which characters can ever share a token. Many emoji are not a single character. A flag or a couple holding hands is built from several Unicode codepoints (numbered characters) joined by an invisible character called the zero-width joiner, written U+200D. To a human it looks like one picture; under the hood, it is a small sequence. `cl100k_base`, used by GPT-4, has no special rule for the joiner. Emoji are not letters, digits, or spaces, so its regex keeps the whole emoji in one chunk. The emoji still costs several tokens, because the vocabulary has few emoji entries. A rainbow flag costs 9 tokens. `o200k_base`, used by GPT-4o, changed the regex so accent-like marks stay with their letters. This helps scripts such as Hindi. The flag costs only 6 tokens there, because the vocabulary is twice as large, not because of a joiner rule. The result is fewer tokens for multilingual text and text with many emoji, which fits GPT-4o's multilingual goals.

**Pre-tokenization** is the regex stage that splits raw text into chunks before BPE merges anything. Its cuts are hard boundaries BPE can never cross. With the vocabulary, it decides how a ZWJ emoji tokenizes. Many emoji are multi-codepoint grapheme clusters. A rainbow flag is a base flag, a variation selector, a zero-width joiner at U+200D, and a rainbow, all rendered as one glyph. The question is whether the regex keeps that cluster together or cuts it apart. Neither regex has a rule for the joiner. Emoji and U+200D are not letters, digits, or spaces, so both patterns match them with the symbol branch. Under cl100k_base the whole flag stays in one chunk. The cost comes from the vocabulary: cl100k has few emoji merges, so the 14 UTF-8 bytes become 9 tokens. The o200k_base regex groups combining marks with letters. The variation selector U+FE0F is a mark, so o200k actually splits this flag into two chunks. It still costs only 6 tokens, because its 200k vocabulary has more byte merges, such as one token for the whole joiner. The emoji savings come from the vocabulary, not the regex. The regex change targets scripts with combining marks: a Hindi word stays in one chunk instead of breaking at every vowel sign. That lowers fertility for non-English text, which fits GPT-4o's multilingual design goals. Measure both encodings with tiktoken rather than assuming.

TL;DR

Neither regex has a ZWJ rule: cl100k_base keeps a ZWJ emoji in one chunk, and o200k_base may even split it at U+FE0F. o200k's lower count (6 vs 9 for 🏳️‍🌈) comes from its larger vocabulary.

Memory aid

Easy to grasp

Picture a sticker made of three smaller stickers taped together with invisible tape: a face, a tape strip, and a heart that together mean one thing. Both the older and the newer system keep the taped sticker in one bag. But the price is set by the price list, and the newer system has a longer price list with cheaper bundles for the pieces. The invisible tape is the zero-width joiner, an unprintable character that glues emoji parts into one symbol, and the price is how many tokens the text costs.

Key concepts

Pre-tokenization regex tiktoken encodings (cl100k vs o200k)ZWJ emoji sequences Multilingual token fertility

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Emoji look like single characters. To a tokenizer, though, they are often short sequences: a base symbol, an invisible joiner, and one or more further symbols that a renderer fuses into one glyph. That gap between what you see and what the bytes contain is why the same rainbow flag can cost nine tokens on one OpenAI model and six on another.

Three layers that are usually discussed separately work together here: Unicode grapheme clusters, the pre-tokenization regex, and the BPE merge constraint. People who think emoji are single codepoints cannot explain the fragmentation. People who think BPE can stitch fragments back together miss the hard boundary rule. People who credit the emoji savings to a regex rule for emoji miss that the vocabulary is the real lever here.

What a ZWJ emoji sequence is in bytes

A surprising number of emoji are not single Unicode codepoints. They are grapheme clusters: several codepoints that a text renderer combines into one visible glyph. The glue is the zero-width joiner, codepoint U+200D, which prints nothing but tells the renderer to fuse its neighbours. A rainbow flag, for example, is a waving white flag, a variation selector that requests emoji presentation, the joiner at U+200D, and a rainbow. That is four codepoints for one glyph.

In UTF-8, each of those codepoints expands to multiple bytes. The joiner alone is three bytes, and the emoji codepoints are four bytes each. So a single rainbow flag is 14 raw bytes before any tokenization decision is made. Skin tone variants, family emoji, and many flags follow the same pattern: a base plus modifiers plus joiners.

The visible glyph is built from a sequence. How many tokens it costs depends on two things: where the regex cuts the sequence, and which byte merges the vocabulary has. Nothing about the picture you see determines that.

How cl100k_base handles the cluster

What o200k_base changed, and where the emoji savings come from

Why OpenAI made the change, and what it means for cost

The limit of the vocabulary: rare emoji still fragment

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

import tiktoken

cl100k = tiktoken.get_encoding("cl100k_base")
o200k = tiktoken.get_encoding("o200k_base")

flag = "\U0001F3F3\uFE0F\u200D\U0001F308"  # rainbow flag (ZWJ sequence)

print(len(cl100k.encode(flag)))  # 9: one chunk, few emoji merges in the vocabulary
print(len(o200k.encode(flag)))   # 6: two chunks, but more byte merges in the vocabulary

# Always resolve the encoding from the model so the regex matches:
enc = tiktoken.encoding_for_model("gpt-4o")  # -> o200k_base

Aspect	cl100k_base (GPT-4)	o200k_base (GPT-4o)
ZWJ emoji sequence	One chunk (symbol branch; no ZWJ rule)	Usually one chunk; split after U+FE0F, which counts as a mark
Tokens for the rainbow flag 🏳️‍🌈	9 (few emoji merges)	6 (more byte merges)
Emoji modifiers (skin tone, flags)	One chunk, many byte-level tokens	One chunk, fewer tokens
Multilingual character runs	Breaks at combining marks (Devanagari vowel signs)	Keeps combining marks with their letters
Design motivation	ASCII-tuned	Multilingual and multimodal

Real products, models, and research that use this idea.

OpenAI's o200k_base, used by GPT-4o and GPT-5, encodes common emoji in fewer tokens than cl100k_base (6 versus 9 for the rainbow flag), mainly because its vocabulary has more byte merges.
Messaging and social products with heavy emoji usage see lower per-message token bills when they move to o200k-based models for the same text.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you estimate the token-cost savings of moving an emoji-heavy chat product from a cl100k model to a GPT-4o model?

Sample real messages, encode under both encodings with tiktoken, weight the per-message token delta by traffic volume.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming o200k_base added a regex rule that keeps ZWJ emoji whole; neither regex mentions U+200D, and o200k's savings on emoji come from its larger vocabulary.

60 second bullets to scan on the way to the call.

What a ZWJ emoji sequence is at the codepoint level
Why pre-tokenization cuts are walls BPE cannot cross

Primary sources. Browse if you want the original framing.