You need to count tokens locally. When do you reach for tiktoken versus SentencePiece?

MCQ·Easy·4.0 · 0·~1 min·Relevant atHugging Face Meta OpenAI

Attempt it

**tiktoken** is OpenAI's fast Rust BPE tokenizer, exposed as a Python package. It ships only the OpenAI encodings: cl100k_base (the GPT-4-era encoder, 100K vocabulary) and o200k_base (the encoder for GPT-4o, o1, o3, GPT-5, and GPT-5.5, with a 200K vocabulary). When you need to count tokens before sending a request to an OpenAI API, you load the matching encoder with tiktoken and run encode on your prompt. **SentencePiece** is Google's tokenizer library. It is written in C++ with Python bindings. It can train and serve both BPE and Unigram LM tokenizers, works directly on raw bytes (no language-specific pre-tokenization required), and produces a single .model artifact. SentencePiece is the tokenizer behind Llama 2, behind Google's Gemma family, behind the T5 family, and behind a long list of multilingual research models. The picking rule is simple. If your model is from OpenAI, use tiktoken. If your model is Gemma, T5, Llama 2, Mistral, or another SentencePiece-based model, use SentencePiece (or HuggingFace's tokenizers wrapper around it). Llama 3 and 4 changed to a tiktoken-style byte-level BPE with a 128K vocabulary, so newer Llama models look more like tiktoken than like SentencePiece.

**tiktoken and SentencePiece** are different tokenizer libraries built by different companies for different model families. They are not interchangeable; running one on text intended for the other gives token counts that are wrong and tokens that the receiving model has never seen. **tiktoken** is OpenAI's fast Rust BPE library, exposed via a Python package of the same name. It ships only the OpenAI encodings as named registry entries. The two encodings that matter in 2026 are cl100k_base (GPT-4, GPT-4 Turbo, GPT-3.5 Turbo; 100K vocab) and o200k_base (GPT-4o, o1, o3, o4-mini, GPT-5, GPT-5.5; 200K vocab). Older encodings (p50k_base, r50k_base) still exist for legacy models. tiktoken is the canonical way to count tokens before an OpenAI API call. **SentencePiece** is Google's tokenizer library, written in C++ with Python bindings. It is a general-purpose toolkit, not a registry of specific encodings. SentencePiece supports both BPE and Unigram LM training, operates on raw bytes without language-specific pre-tokenization (so no separate word-splitting step is needed for languages like Japanese or Thai), and produces a single .model file plus an optional .vocab file. SentencePiece ships as the tokenizer for Llama 2, Gemma, T5, FLAN-T5, and a long tail of multilingual research models. The selection rule has three branches. OpenAI models: tiktoken. Llama 2, Gemma, T5, and SentencePiece-trained models: SentencePiece directly or via HuggingFace tokenizers. Llama 3 and 4: tiktoken-style byte-level BPE with a 128K vocabulary, accessed via HuggingFace tokenizers (not via tiktoken itself, because the OpenAI tiktoken package does not ship the Llama encoder).

Quick reference

TL;DR

tiktoken is OpenAI's BPE-only tokenizer with the cl100k_base and o200k_base encodings; SentencePiece is Google's general tokenizer library that supports BPE and Unigram LM and ships with Gemma, T5, and Llama 2.

Memory aid

Easy to grasp

Think of tiktoken as a very fast cashier who only knows the prices at one specific store. If you walk into that store (OpenAI) it is the quickest, friendliest way to ring up your bill. If you walk into a different store, the cashier has no idea what anything costs, because they only know the OpenAI catalog. SentencePiece is more like a general-purpose cash-register kit. You can set it up for many different stores. Google built it, and it is the register used by Gemma, by T5, and by older Llama 2 models. Newer Llama 3 and 4 moved to a register that mimics the OpenAI style, but the original was SentencePiece. The picking rule is simple. If you are calling an OpenAI API, you reach for tiktoken because you need the OpenAI encodings to count tokens correctly. For pretty much anything else, you reach for SentencePiece or the HuggingFace tokenizers wrapper around it.

Key concepts

tiktoken SentencePiececl100k_base Encodingo200k_base Encoding

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: identify tiktoken as OpenAI's BPE library with cl100k_base and o200k_base; identify SentencePiece as Google's general toolkit supporting BPE and Unigram LM; name three production models on each side; explain the Llama 3 switch from SentencePiece-Unigram to a tiktoken-style BPE; close with the rule that mixing them silently is a common bug source.

Real-world examples

Real products, models, and research that use this idea.

Counting tokens for a GPT-5.5 chat request: import tiktoken, get_encoding('o200k_base'), encode the prompt, take the length, compare to the model's context window.
Loading the Gemma 3 tokenizer: SentencePieceProcessor().Load('gemma_tokenizer.model'); the .model file contains both vocabulary and merge or pruning state.
Loading Llama 3.3 tokenizer via HuggingFace: AutoTokenizer.from_pretrained('meta-llama/Llama-3.3-70B-Instruct'); the tokenizer.json artifact uses a tiktoken-style byte-level BPE with a 128K vocabulary.
Counting Claude tokens: the proprietary tokenizer is not shipped as a library; the canonical way is anthropic.Anthropic().messages.count_tokens with the model name.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does tiktoken make encoding so fast compared to a pure-Python BPE implementation?

QWhy does SentencePiece bother to operate on raw bytes instead of pre-tokenizing on whitespace like classical tokenizers?

QLlama 3 uses a tiktoken-style BPE but is not in tiktoken; how do you load it?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating tiktoken and SentencePiece as interchangeable token counters. They are different tokenizers with different vocabularies; running tiktoken on a Llama prompt and trusting the count is just wrong.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.