A developer saves API calls by tokenizing Llama-3 prompts client side using tiktoken with cl100k_base for cost estimation, then sends the same text to a Llama-3 API endpoint. Separately, they try to use tiktoken encoded token IDs to index directly into a Llama embedding matrix. Describe what breaks in each case and why.
Llama-3's tokenizer is cl100k_base plus 28K added tokens, so cl100k counts drift on non-English text and its IDs line up only on the shared 100K, breaking special tokens and the added range.
Imagine two libraries that each give books their own shelf numbers. Book #100257 in one library might be a cookbook, while #100257 in the other is a poetry collection. The numbers look the same but point to totally different things. A tokenizer is like that shelving system: it turns text into numbered slots. The Llama-3 system copied its first 100,000 slot numbers from tiktoken, which is OpenAI's system, then added 28,000 new slots, mostly for other languages. So if you count slots with one library's catalog to guess how big a job is in the other, your guess drifts on those languages. And if you hand over OpenAI's slot numbers, the copied slots come back right, but the new slots are never used and OpenAI's special numbers fetch the wrong books.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Tokenizers feel like a neutral utility: text in, integers out. The trap is assuming those integers mean the same thing everywhere. They do not. Each model family trains its own tokenizer, and the resulting vocabulary is as much a part of the model as its weights.
Consider two tempting shortcuts. One reuses OpenAI's tokenizer to estimate Llama costs. The other reuses its token IDs to drive Llama's embeddings. Both come from the same misconception, but they fail with very different severity. That difference in severity matters most.
tiktoken the engine versus cl100k_base the vocabulary
First, untangle the names. tiktoken is OpenAI's fast Rust-backed BPE tokenizer library. It is not a single mapping; it loads named encodings. cl100k_base is one such encoding, used by GPT-4-class models, with roughly 100k entries. Newer GPT-5.x models use o200k_base, a different and larger vocabulary.
Llama-3 ships its own tokenizer, distributed through its model repository, with a vocabulary near 128k entries. It keeps the 100K cl100k_base tokens and adds 28K tokens to better support non-English languages.
Both vocabularies are trained. A BPE vocabulary is learned by counting frequent adjacent pairs in a specific corpus and merging them. Two corpora, even if both are mostly English, produce different merge orders and therefore different tokens. Llama-3 is a special case: it agrees with cl100k_base on the shared tokens and their integers, but its added merges change the split for many non-English words, and its special tokens differ. Treating them as interchangeable is the root error behind both failure cases. It is an easy error to make because the two tokenizers produce identical lists of integers for simple English.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# Wrong: using tiktoken to count Llama-3 tokens
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
prompt = "Explain tokenization briefly."
print("cl100k estimate:", len(enc.encode(prompt))) # matches on English, off on non-English
# Right: use Llama's own tokenizer
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
print("Llama-3 actual:", len(tok.encode(prompt))) # matches cost
# Wrong: cl100k IDs into Llama's embedding matrix
cl100k_ids = enc.encode(prompt) # IDs for the cl100k vocab
# llama_model.embed_tokens(torch.tensor([cl100k_ids])) # special and non-English tokens wrongReal products, models, and research that use this idea.
- OpenAI's tiktoken ships cl100k_base for earlier GPT models and o200k_base for the GPT-5.x family, each a distinct ID space.
- Meta's Llama 4 Maverick loads its own tokenizer from the HuggingFace repo; counting its tokens requires that tokenizer, not tiktoken.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build a cost estimator that stays accurate across English, code, and non-Latin scripts for a given model?
Load the target model's actual tokenizer and encode the real payload; avoid char to token heuristics and per-language correction factors that drift on mixed content.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming token IDs are a universal standard. Each tokenizer maps its own integers to its own strings, so the same integer means different text in different vocabularies.
60 second bullets to scan on the way to the call.
Why two tokenizers split the same text into different token counts
What fertility means and how it varies by content type
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.