You need to count tokens locally. When do you reach for tiktoken versus SentencePiece?

MCQ·Easy·4.0 · 0·~1 min·Asked atLinkedin Replicate Uniphore·Relevant atGoogle Hugging Face Meta OpenAI

Attempt it

**tiktoken** is OpenAI's fast Rust BPE tokenizer, exposed as a Python package. It ships only the OpenAI encodings: cl100k_base (the GPT-4-era encoder, 100K vocabulary) and o200k_base (the encoder for GPT-4o, o1, o3, GPT-5, and GPT-5.5, with a 200K vocabulary). When you need to count tokens before sending a request to an OpenAI API, you load the matching encoder with tiktoken and run encode on your prompt. **SentencePiece** is Google's tokenizer library. It is written in C++ with Python bindings. It can train and serve both BPE and Unigram LM tokenizers, works directly on raw bytes (no language-specific pre-tokenization required), and produces a single .model artifact. SentencePiece is the tokenizer behind Llama 2, behind Google's Gemma family, behind the T5 family, and behind a long list of multilingual research models. The picking rule is simple. If your model is from OpenAI, use tiktoken. If your model is Gemma, T5, Llama 2, Mistral, or another SentencePiece-based model, use SentencePiece (or HuggingFace's tokenizers wrapper around it). Llama 3 and 4 changed to a tiktoken-style byte-level BPE with a 128K vocabulary, so newer Llama models look more like tiktoken than like SentencePiece.

**tiktoken and SentencePiece** are different tokenizer libraries built by different companies for different model families. They are not interchangeable; running one on text intended for the other gives token counts that are wrong and tokens that the receiving model has never seen. **tiktoken** is OpenAI's fast Rust BPE library, exposed via a Python package of the same name. It ships only the OpenAI encodings as named registry entries. The two encodings that matter in 2026 are cl100k_base (GPT-4, GPT-4 Turbo, GPT-3.5 Turbo; 100K vocab) and o200k_base (GPT-4o, o1, o3, o4-mini, GPT-5, GPT-5.5; 200K vocab). Older encodings (p50k_base, r50k_base) still exist for legacy models. tiktoken is the canonical way to count tokens before an OpenAI API call. **SentencePiece** is Google's tokenizer library, written in C++ with Python bindings. It is a general-purpose toolkit, not a registry of specific encodings. SentencePiece supports both BPE and Unigram LM training, operates on raw bytes without language-specific pre-tokenization (so no separate word-splitting step is needed for languages like Japanese or Thai), and produces a single .model file plus an optional .vocab file. SentencePiece ships as the tokenizer for Llama 2, Gemma, T5, FLAN-T5, and a long tail of multilingual research models. The selection rule has three branches. OpenAI models: tiktoken. Llama 2, Gemma, T5, and SentencePiece-trained models: SentencePiece directly or via HuggingFace tokenizers. Llama 3 and 4: tiktoken-style byte-level BPE with a 128K vocabulary, accessed via HuggingFace tokenizers (not via tiktoken itself, because the OpenAI tiktoken package does not ship the Llama encoder).

TL;DR

tiktoken is OpenAI's BPE-only tokenizer with the cl100k_base and o200k_base encodings; SentencePiece is Google's general tokenizer library that supports BPE and Unigram LM and ships with Gemma, T5, and Llama 2.

Memory aid

Easy to grasp

Think of tiktoken as a very fast cashier who only knows the prices at one specific store. If you walk into that store (OpenAI) it is the quickest, friendliest way to ring up your bill. If you walk into a different store, the cashier has no idea what anything costs, because they only know the OpenAI catalog. SentencePiece is more like a general-purpose cash-register kit. You can set it up for many different stores. Google built it, and it is the register used by Gemma, by T5, and by older Llama 2 models. Newer Llama 3 and 4 moved to a register that mimics the OpenAI style, but the original was SentencePiece. The picking rule is simple. If you are calling an OpenAI API, you reach for tiktoken because you need the OpenAI encodings to count tokens correctly. For pretty much anything else, you reach for SentencePiece or the HuggingFace tokenizers wrapper around it.

Key concepts

tiktoken SentencePiececl100k_base Encodingo200k_base Encoding

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

tiktoken and SentencePiece are the two tokenizer libraries that most often come up in 2026 production work, and confusing them is one of the easier bugs to introduce in a multi-model pipeline. They serve different model families, support different sets of algorithms, and produce token ids that mean nothing to each other.

This deep dive separates the two libraries by purpose, lists which 2026 models use which, names a third library (HuggingFace tokenizers) that increasingly mediates between them, and finishes with the practical selection rule.

tiktoken: OpenAI's narrow, fast registry

tiktoken is built for exactly one job: tokenize text for OpenAI models, quickly, with the exact same vocabulary the API server uses. The Python package wraps a Rust core that handles the regex-based pre-tokenization and byte-level BPE encoding. Encoding a million tokens runs in the hundreds of milliseconds range rather than the tens of seconds range that a pure-Python BPE would take.

The library's surface is small. There is a registry of named encodings, and you load one by name. The two that matter in 2026 are:

cl100k_base: 100K vocabulary, used by GPT-4, GPT-4 Turbo, and GPT-3.5 Turbo. Still relevant because many production pipelines have not migrated off these models.
o200k_base: 200K vocabulary, used by GPT-4o, o1, o3, o4-mini, GPT-5, and GPT-5.5. This is the default for new OpenAI work in 2026.

Older encodings (p50k_base, r50k_base) exist for legacy Codex and GPT-3 models and rarely show up in greenfield work.

tiktoken cannot train tokenizers, cannot tokenize text for non-OpenAI models, and does not ship the Llama, Gemma, or Claude vocabularies. It is a closed registry exposed for one purpose: counting tokens accurately before paying per-token API calls.

SentencePiece: Google's general toolkit

The Llama 3 inflection point

HuggingFace tokenizers: the production wrapper

The selection rule and why mixing them silently is a common bug

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Counting tokens for a GPT-5.5 chat request: import tiktoken, get_encoding('o200k_base'), encode the prompt, take the length, compare to the model's context window.
Loading the Gemma 3 tokenizer: SentencePieceProcessor().Load('gemma_tokenizer.model'); the .model file contains both vocabulary and merge or pruning state.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does tiktoken make encoding so fast compared to a pure-Python BPE implementation?

tiktoken's core is written in Rust and exposed to Python via PyO3. The hot path (regex pre-tokenization, byte-level BPE merging, hash-table lookups for token ids) runs in compiled native code, so encoding a million tokens takes hundreds of milliseconds rather than tens of seconds. The Python layer is a thin wrapper that mostly handles the registry of named encodings.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating tiktoken and SentencePiece as interchangeable token counters. They are different tokenizers with different vocabularies; running tiktoken on a Llama prompt and trusting the count is just wrong.

60 second bullets to scan on the way to the call.

Identify tiktoken as OpenAI's Rust BPE library exposed as a Python package.
Name the two OpenAI encodings that matter in 2026 and which models use each.

Primary sources. Browse if you want the original framing.