What would break if you tokenized a Llama prompt using tiktoken's cl100k_base instead of the Llama tokenizer?

Attempt it

A developer saves API calls by tokenizing Llama-3 prompts client-side using tiktoken with cl100k_base for cost estimation, then sends the same text to a Llama-3 API endpoint. Separately, they try to use tiktoken-encoded token IDs to index directly into a Llama embedding matrix. Describe what breaks in each case and why.

Free · 2 AI evals / day

Quick reference

TL;DR

tiktoken and the Llama tokenizer have different vocabularies, so the same text yields different token counts (bad cost estimates) and the IDs are not portable (garbage embedding lookups).

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine two libraries that each give books their own shelf numbers. Book #4891 in one library might be a cookbook, while #4891 in the other is a poetry collection. The numbers look the same but point to totally different things. A tokenizer is like that shelving system: it turns text into numbered slots. tiktoken, which is OpenAI's system, and the Llama system number their slots differently. So if you count slots with one library's catalog to guess how big a job is in the other, your guess drifts. And if you hand one library's shelf numbers to the other and ask it to fetch the matching books, it grabs whatever happens to sit at those numbers, which is the wrong content every time.

Key concepts

tiktoken Embedding matrix

Quick scan · 2–4 min

Everything important, quickly.

3 min: separate vocabularies + fertility differs by content + IDs are vocab-specific pointers + wrong embedding rows or out of range + bind tokenizer to model.

Code example

python

# Wrong: using tiktoken to count Llama-3 tokens
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
prompt = "Explain tokenization briefly."
print("cl100k estimate:", len(enc.encode(prompt)))  # off vs Llama

# Right: use Llama's own tokenizer
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
print("Llama-3 actual:", len(tok.encode(prompt)))    # matches cost

# Fatal: cl100k IDs into Llama's embedding matrix
cl100k_ids = enc.encode(prompt)   # IDs for the cl100k vocab
# llama_model.embed_tokens(torch.tensor([cl100k_ids]))  # wrong rows!

Real-world examples

Real products, models, and research that use this idea.

OpenAI's tiktoken ships cl100k_base for earlier GPT models and o200k_base for the GPT-5.x family, each a distinct ID space.
Meta's Llama 4 Maverick loads its own tokenizer from the HuggingFace repo; counting its tokens requires that tokenizer, not tiktoken.
LangChain and LlamaIndex token counters let you select the tokenizer per model so cost estimates match the target endpoint's billing.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you build a cost estimator that stays accurate across English, code, and non-Latin scripts for a given model?

Sign in or go Pro to see the approach for this follow-up.

QIf you had to map text encoded by cl100k_base into a Llama model, what would a correct bridge look like?

Sign in or go Pro to see the approach for this follow-up.

QWhy can't you just retrain Llama's embedding rows to align with cl100k_base IDs?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.