Anthropic does not publish Claude's tokenizer; use the messages.count_tokens API for exact counts. tiktoken, len/4 heuristics, and the deprecated Claude 2 tokenizer are all wrong for Claude 3 and later.
Imagine four ways to weigh a package before mailing it. One is to use the post office's official scale, which gives the exact weight you will be charged. The other three are to use a friend's bathroom scale, to guess by lifting, or to use an old kitchen scale you found in storage. Only the official scale matches the bill. The same logic applies to token counts for Claude. Anthropic does not let you bring your own scale because they have not published the tokenizer. They do let you ask their server how many tokens a message will be, and that answer is what you get billed for. Every other method is a guess, and the guesses can be 10 to 30 percent off, which is a lot when token counts drive both cost and context-window truncation.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Token counting looks like a solved problem until you discover that the right method depends on which provider you are billing. There is no universal tokenizer. There is no library that gives correct counts for everyone. The candidate who treats this as a single technical question with a single answer misses the structural issue, which is that tokenization is part of the model and each provider exposes a different interface for getting an exact count.
The correct answer for Claude Sonnet 4.6 in 2026 is messages.count_tokens. Anthropic does not publish the tokenizer, so local computation is not possible. The API endpoint is the only path to a byte-exact count. The three wrong options all reuse tools from other providers or other Claude versions and are systematically wrong.
The rest of this explanation walks through why each wrong option fails, why Anthropic's design choice makes sense, what the per-provider rule looks like in production, and what the operational realities are for teams that have to count tokens at scale across multiple providers.
Why tiktoken on Claude is wrong
tiktoken is OpenAI's open-source BPE tokenizer library, with cl100k_base for GPT-4 and o200k_base for GPT-4o, GPT-5, and the o-series. These are the actual tokenizers OpenAI's models use, so the counts are byte-exact for OpenAI.
For Claude, the tokenizer is a different BPE with a different merge table. Same algorithm family, completely different vocabulary. The empirical drift between tiktoken counts and actual Claude counts is 10 to 30 percent depending on the content. English prose tends to drift less; code drifts more in the OpenAI-undercount direction; non-English text drifts unpredictably.
A team using tiktoken to estimate Claude costs is making a systematic error of that magnitude. If your cost model says 'we will spend $10,000 per month on Claude', the actual bill is between $7,000 and $13,000. If your budget enforcement uses the local estimate to decide when to truncate a context, you will truncate at the wrong byte every time. Neither of these is acceptable for production.
The deeper structural error is treating tokenization as portable. The tokenizer is part of the model. You cannot mix and match. Anyone shipping a tokenizer choice that does not match the model is one outage away from learning this lesson the hard way.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# Right way: Anthropic messages.count_tokens
from anthropic import Anthropic
client = Anthropic()
messages = [
{'role': 'user', 'content': 'Summarize this contract: ...'}
]
resp = client.messages.count_tokens(
model='claude-sonnet-4-6',
system='You are a legal summarization assistant.',
messages=messages,
tools=my_tool_defs, # tool schema counts too
)
print(resp.input_tokens) # exact, matches billing
# Wrong: tiktoken on a Claude prompt
import tiktoken
enc = tiktoken.get_encoding('cl100k_base')
print(len(enc.encode('Summarize this contract: ...'))) # 10-30% off for Claude
# Wrong: len/4 heuristic on code
print(len(open('file.py').read()) // 4) # underestimates badly on codeReal products, models, and research that use this idea.
- Anthropic's API reference documents messages.count_tokens as the supported method for Claude 3 and the Claude 4 family, accepting the same payload as messages.create and returning input_tokens.
- OpenAI's tiktoken library ships encoding_for_model() that returns cl100k_base for GPT-4 and o200k_base for GPT-4o, GPT-5, and o-series models.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour service makes 50 LLM calls per second to Claude. Can you afford to call count_tokens before every one?
Probably not on the hot path. Two mitigations: cache by content hash for repeated prompts (system prompts, templates) so count_tokens runs once per unique payload, and batch count_tokens calls for asynchronous cost reporting. For real-time budget enforcement, use a conservative local upper-bound estimate and reconcile against actual billed tokens from response.usage.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reusing tiktoken cl100k_base or the deprecated Claude 2 tokenizer to estimate Claude 3 token counts, then trusting the number for billing or context-budget logic.
60 second bullets to scan on the way to the call.
Why Anthropic does not let you tokenize locally for Claude.
What the messages.count_tokens endpoint returns and what fields it accepts.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.