**o200k_base** is the tokenizer that ships with GPT-4o and newer OpenAI models. The '200k' is its vocabulary size: about 200,000 distinct tokens. A tokenizer's vocabulary is the fixed dictionary of text pieces it knows, and any input gets chopped into entries from that dictionary.
The older `cl100k_base`, used by GPT-4 and GPT-3.5-turbo, has roughly 100,000 entries. By doubling the dictionary, o200k_base gives dedicated single-token slots to many patterns that cl100k had to split into several pieces, especially non-English words, Chinese and Japanese characters, and emoji.
The team also refreshed the pre-tokenization regex, the rule that decides where text is first chopped before merging. The combined effect is that the same Japanese sentence or emoji string becomes noticeably fewer tokens. Fewer tokens means lower cost and more room in the context window for non-English users.
The '200k' in **o200k_base** is the vocabulary size: roughly 200,000 tokens, double the ~100,000 in `cl100k_base`. It is not a context window and not a byte count. It is the size of the tokenizer's dictionary.
Two things improved. First, the larger vocabulary lets common non-English patterns claim their own single token instead of splitting into byte fragments. Chinese, Japanese, and emoji especially benefit, because cl100k spent most of its budget on English. Second, the pre-tokenization regex was updated to handle Unicode ZWJ emoji sequences and multilingual character runs more gracefully.
The practical payoff is fewer tokens for the same multilingual text. That directly lowers cost and stretches the effective context window.
$$\text{cost} = \text{tokens} \times \text{price-per-1k}$$
Cost scales with token count. So a tokenizer that emits fewer tokens for Japanese or emoji-heavy prompts is cheaper per request even at the same per-token price. The catch is that a 200k vocabulary means a larger embedding matrix and `lm_head`, so the win is paid for in model parameters.
Quick reference
TL;DR
o200k_base doubles the vocabulary to ~200k tokens with a refreshed regex, so multilingual and emoji-heavy text tokenizes into far fewer tokens than under cl100k_base.
Imagine you label moving boxes with sticky notes. With only 100 notes you give the boxes you pack most often (English words) their own label, but rarer boxes (Japanese, emoji) have to share. Now picture getting 200 notes. A lot of those once-shared boxes suddenly get their own label, so you write fewer notes overall to describe the same pile. That is what o200k_base does: it is the newer OpenAI vocabulary with twice as many labels as cl100k_base, so non-English text and emoji get packed into fewer pieces.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Counting tokens for a GPT-4 or GPT-3.5-turbo call Those models use cl100k_base, so o200k counts would misestimate cost and context headroom.
Designing a small on-device model with tight memory A 200k vocabulary inflates the embedding matrix and lm_head, which can dominate a small model's parameters.
Workloads that are almost entirely short English text English was already well covered by cl100k, so the o200k savings are marginal there.
Quick scan · 2–4 min
Everything important, quickly.
3 min: vocabulary 100k vs 200k + where the budget shifts (CJK, emoji) + regex refresh + cost = tokens x price + embedding/lm_head tradeoff + sticky encoding per model.
Code example
python
import tiktoken
cl100k = tiktoken.get_encoding("cl100k_base")
o200k = tiktoken.get_encoding("o200k_base")
text = "今日はいい天気です。" # "It's nice weather today." in Japanese
print(len(cl100k.encode(text))) # more tokens (English-biased vocab)
print(len(o200k.encode(text))) # fewer tokens (larger CJK coverage)
# Always resolve the encoding from the model, never hardcode:
enc = tiktoken.encoding_for_model("gpt-4o") # -> o200k_base
Side-by-side
Property
cl100k_base
o200k_base
Vocabulary size
~100,000 tokens
~200,000 tokens
Model family
GPT-4, GPT-3.5-turbo
GPT-4o, GPT-5.5
Multilingual / CJK efficiency
English-biased, higher fertility
Lower fertility, fewer tokens
Pre-tokenization regex
Older pattern
Updated for ZWJ emoji and multilingual runs
Embedding / lm_head cost
Smaller
Larger (vocabulary scales both)
Real-world examples
Real products, models, and research that use this idea.
1OpenAI ships o200k_base with the current GPT-5.5 family, while the older GPT-3.5-turbo line stays on cl100k_base.
2The tiktoken library exposes both encodings, and tiktoken.encoding_for_model() returns o200k_base for current OpenAI flagship models.
3Multilingual products serving CJK users pick the o200k-based models specifically because the same prompt costs fewer tokens than under cl100k.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you verify the multilingual token savings before migrating a production pipeline to a GPT-4o model?
Sign in or go Pro to see the approach for this follow-up.
QWhy can't you just retrofit o200k onto an existing cl100k model to get the savings?
Sign in or go Pro to see the approach for this follow-up.
QWhen might the larger vocabulary actually hurt you?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reading the '200k' as a 200,000-token context window rather than a 200,000-entry vocabulary: the number describes the tokenizer, not the prompt budget.