The correct answer is **B: call Anthropic's messages.count_tokens endpoint**.
Anthropic does not publish a tokenizer library for Claude. Unlike OpenAI, which ships tiktoken and lets you compute token counts locally, Anthropic keeps Claude's tokenizer internal. The only supported way to get an exact token count for a Claude prompt is to ask Anthropic's server. The `messages.count_tokens` endpoint takes the same arguments as the `messages.create` endpoint (system prompt, message list, tools) and returns the exact `prompt_tokens` you will be billed for. It was released in beta in 2024 and went GA in 2025.
The wrong options each fail in a specific way. **A** (tiktoken cl100k_base) is the GPT-4 tokenizer and produces counts that are off by 10 to 30 percent for Claude, because the vocabularies are completely different. **C** (len/4 heuristic) is a rough thumb for English prose; it breaks badly on code (more tokens), JSON (lots of punctuation), and non-English text (much higher fertility). **D** (Claude 2 tokenizer) was specific to Claude 1 and 2 and is deprecated; using it for Claude 3 gives inaccurate counts.
The per-provider rule: tiktoken for OpenAI, messages.count_tokens for Claude, the model's own HuggingFace tokenizer for open-weight models like Llama 3 and Mistral.
**B: messages.count_tokens.**
Anthropic does not publish Claude's tokenizer. The supported method for exact token counts on Claude 3 onwards (including the 4.x family) is the `messages.count_tokens` API endpoint, released in beta in 2024 and GA in 2025. It accepts the same payload as `messages.create` (system, messages, tools) and returns `input_tokens`, matching what billing will record. This is the only method that is byte-exact for Claude.
**Why the other options fail.**
**A (tiktoken cl100k_base):** cl100k_base is the GPT-4 BPE. Claude's tokenizer is a different, undisclosed BPE with a different merge table. Empirical drift is 10 to 30 percent depending on content, larger for code and non-English text. Using it for cost modeling or context-window budgeting produces systematic error.
**C (len/4 heuristic):** Approximately 4 characters per token holds for English prose on most modern tokenizers, but breaks on code (closer to 3), JSON with many short keys (closer to 2.5), and non-English text (Hindi can be 2-3 characters per token, inverting the ratio). Heuristics are fine for back of the envelope estimates but unsafe for production logic.
**D (Claude 2 tokenizer):** Anthropic shipped a tokenizer package for Claude 1 and 2 that is deprecated and inaccurate for Claude 3 and later. The vocabularies changed between generations.
**Per-provider rule:** tiktoken (`get_encoding('cl100k_base')` or `'o200k_base'`) for OpenAI, `messages.count_tokens` for Claude, `AutoTokenizer.from_pretrained(...)` for open-weight models. Google Gemini bills per character rather than per token, which adds its own wrinkle when modeling cost across providers.
Quick reference
TL;DR
Anthropic does not publish Claude's tokenizer; use the messages.count_tokens API for exact counts. tiktoken, len/4 heuristics, and the deprecated Claude 2 tokenizer are all wrong for Claude 3 and later.
Imagine four ways to weigh a package before mailing it. One is to use the post office's official scale, which gives the exact weight you will be charged. The other three are to use a friend's bathroom scale, to guess by lifting, or to use an old kitchen scale you found in storage. Only the official scale matches the bill. The same logic applies to token counts for Claude. Anthropic does not let you bring your own scale because they have not published the tokenizer. They do let you ask their server how many tokens a message will be, and that answer is what you get billed for. Every other method is a guess, and the guesses can be 10 to 30 percent off, which is a lot when token counts drive both cost and context-window truncation.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A pure back of the envelope estimate during early prototyping when nothing is being billed yet. len/4 or tiktoken approximations are fine when the error has no consequence; the discipline matters once the count drives a cost model or a budget gate.
Counting tokens for the Gemini family where the billing primitive is characters, not tokens. Token counting is the wrong unit; use character counts that match Gemini's billing model and reserve token math for context-window logic only.
A high-QPS path where calling count_tokens for every request would exceed rate limits. Cache by content hash or fall back to a calibrated local estimate; the API call has rate limits even when free.
Quick scan · 2–4 min
Everything important, quickly.
Open with the framing that the tokenizer is part of the model and token counting is provider-specific. Name messages.count_tokens as the right answer for Claude 3 and 3.7. Dismantle each distractor: tiktoken is the wrong BPE (10-30 percent error), len/4 breaks on code and non-English, Claude 2 tokenizer is deprecated. Close with the per-provider rule (tiktoken for OpenAI, count_tokens for Claude, AutoTokenizer for open-weight, character-count for Gemini) and the production wrinkle around tool definitions counting toward the same budget.
Code example
python
# Right way: Anthropic messages.count_tokens
from anthropic import Anthropic
client = Anthropic()
messages = [
{'role': 'user', 'content': 'Summarize this contract: ...'}
]
resp = client.messages.count_tokens(
model='claude-sonnet-4-6',
system='You are a legal summarization assistant.',
messages=messages,
tools=my_tool_defs, # tool schema counts too
)
print(resp.input_tokens) # exact, matches billing
# Wrong: tiktoken on a Claude prompt
import tiktoken
enc = tiktoken.get_encoding('cl100k_base')
print(len(enc.encode('Summarize this contract: ...'))) # 10-30% off for Claude
# Wrong: len/4 heuristic on code
print(len(open('file.py').read()) // 4) # underestimates badly on code
Real-world examples
Real products, models, and research that use this idea.
1Anthropic's API reference documents messages.count_tokens as the supported method for Claude 3 and the Claude 4 family, accepting the same payload as messages.create and returning input_tokens.
2OpenAI's tiktoken library ships encoding_for_model() that returns cl100k_base for GPT-4 and o200k_base for GPT-4o, GPT-5, and o-series models.
3Google Gemini's billing page in 2026 lists character-based pricing for input and output, distinct from token-based pricing on OpenAI and Anthropic.
4HuggingFace transformers AutoTokenizer.from_pretrained loads the exact tokenizer that ships with Llama 3, Mistral, and Qwen 3.5 checkpoints, giving byte-exact counts for open-weight models.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QYour service makes 50 LLM calls per second to Claude. Can you afford to call count_tokens before every one?
Sign in or go Pro to see the approach for this follow-up.
QIf you have to estimate Claude token counts locally without count_tokens, what's the least-bad approach?
Sign in or go Pro to see the approach for this follow-up.
QWhy does Google Gemini bill per character instead of per token, and what does that change for cost modeling?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reusing tiktoken cl100k_base or the deprecated Claude 2 tokenizer to estimate Claude 3 token counts, then trusting the number for billing or context-budget logic.