A developer saves API calls by tokenizing Llama-3 prompts client side using tiktoken with cl100k_base for cost estimation, then sends the same text to a Llama-3 API endpoint. Separately, they try to use tiktoken encoded token IDs to index directly into a Llama embedding matrix. Describe what breaks in each case and why.
tiktoken and the Llama tokenizer have different vocabularies, so the same text yields different token counts (bad cost estimates) and the IDs are not portable (garbage embedding lookups).
Imagine two libraries that each give books their own shelf numbers. Book #4891 in one library might be a cookbook, while #4891 in the other is a poetry collection. The numbers look the same but point to totally different things. A tokenizer is like that shelving system: it turns text into numbered slots. tiktoken, which is OpenAI's system, and the Llama system number their slots differently. So if you count slots with one library's catalog to guess how big a job is in the other, your guess drifts. And if you hand one library's shelf numbers to the other and ask it to fetch the matching books, it grabs whatever happens to sit at those numbers, which is the wrong content every time.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Tokenizers feel like a neutral utility: text in, integers out. The trap is assuming those integers mean the same thing everywhere. They do not. Each model family trains its own tokenizer, and the resulting vocabulary is as much a part of the model as its weights.
This question puts two tempting shortcuts side by side. One reuses OpenAI's tokenizer to estimate Llama costs; the other reuses its token IDs to drive Llama's embeddings. Both come from the same misconception, but they fail with very different severity, and that difference in severity is the real thing the interviewer is probing.
We will separate tiktoken the encoder from cl100k_base the vocabulary, walk through why counts diverge, then show why IDs are pointers that only make sense inside one table. After that we cover the correct bridge between vocabularies and how to detect this class of bug before it reaches production. By the end the difference between a soft cost error and a hard correctness bug should be obvious.
tiktoken the engine versus cl100k_base the vocabulary
First, untangle the names. tiktoken is OpenAI's fast Rust-backed BPE tokenizer library. It is not a single mapping; it loads named encodings. cl100k_base is one such encoding, used by GPT-4-class and GPT-4o models, with roughly 100k entries. Newer GPT-5.x models use o200k_base, a different and larger vocabulary.
Llama ships its own tokenizer entirely, distributed through its model repository, with a vocabulary near 128k entries. It was trained on Meta's corpus with its own merge priorities.
The key word is trained. A BPE vocabulary is learned by counting frequent adjacent pairs in a specific corpus and merging them. Two corpora, even both English-heavy, produce different merge orders and therefore different tokens. So cl100k_base and Llama's tokenizer disagree both on which substrings become single tokens and on what integer each token gets. They also differ in pre-tokenization rules, special tokens, and how whitespace attaches to words, all of which shift the output further apart. Treating them as interchangeable is the root error behind both failure cases, and it is an easy error to make precisely because the two tokenizers produce superficially similar lists of integers for simple English.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# Wrong: using tiktoken to count Llama-3 tokens
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
prompt = "Explain tokenization briefly."
print("cl100k estimate:", len(enc.encode(prompt))) # off vs Llama
# Right: use Llama's own tokenizer
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
print("Llama-3 actual:", len(tok.encode(prompt))) # matches cost
# Fatal: cl100k IDs into Llama's embedding matrix
cl100k_ids = enc.encode(prompt) # IDs for the cl100k vocab
# llama_model.embed_tokens(torch.tensor([cl100k_ids])) # wrong rows!Real products, models, and research that use this idea.
- OpenAI's tiktoken ships cl100k_base for earlier GPT models and o200k_base for the GPT-5.x family, each a distinct ID space.
- Meta's Llama 4 Maverick loads its own tokenizer from the HuggingFace repo; counting its tokens requires that tokenizer, not tiktoken.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build a cost estimator that stays accurate across English, code, and non-Latin scripts for a given model?
Load the target model's actual tokenizer and encode the real payload; avoid char to token heuristics and per-language correction factors that drift on mixed content.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming token IDs are a universal standard. Each tokenizer maps its own integers to its own strings, so the same integer means different text in different vocabularies.
60 second bullets to scan on the way to the call.
Why two tokenizers split the same text into different token counts
What fertility means and how it varies by content type
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.