You need the exact token count for a Claude Sonnet 4.6 prompt before sending it. Which method actually works?
Anthropic does not publish Claude's tokenizer; use the messages.count_tokens API for exact counts. tiktoken, len/4 heuristics, and the deprecated Claude 2 tokenizer are all wrong for Claude 3 and later.
Imagine four ways to weigh a package before mailing it. One is to use the post office's official scale, which gives the exact weight you will be charged. The other three are to use a friend's bathroom scale, to guess by lifting, or to use an old kitchen scale you found in storage. Only the official scale matches the bill. The same logic applies to token counts for Claude. Anthropic does not let you bring your own scale because they have not published the tokenizer. They do let you ask their server how many tokens a message will be, and that answer is what you get billed for. Every other method is a guess, and the guesses can be 10 to 30 percent off, which is a lot when token counts drive both cost and context-window truncation.
Detailed answer & concept explanation~7 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Open with the framing that the tokenizer is part of the model and token counting is provider-specific. Name messages.count_tokens as the right answer for Claude 3 and 3.7. Dismantle each distractor: tiktoken is the wrong BPE (10-30 percent error), len/4 breaks on code and non-English, Claude 2 tokenizer is deprecated. Close with the per-provider rule (tiktoken for OpenAI, count_tokens for Claude, AutoTokenizer for open-weight, character-count for Gemini) and the production wrinkle around tool definitions counting toward the same budget.
# Right way: Anthropic messages.count_tokens
from anthropic import Anthropic
client = Anthropic()
messages = [
{'role': 'user', 'content': 'Summarize this contract: ...'}
]
resp = client.messages.count_tokens(
model='claude-sonnet-4-6',
system='You are a legal summarization assistant.',
messages=messages,
tools=my_tool_defs, # tool schema counts too
)
print(resp.input_tokens) # exact, matches billing
# Wrong: tiktoken on a Claude prompt
import tiktoken
enc = tiktoken.get_encoding('cl100k_base')
print(len(enc.encode('Summarize this contract: ...'))) # 10-30% off for Claude
# Wrong: len/4 heuristic on code
print(len(open('file.py').read()) // 4) # underestimates badly on codeReal products, models, and research that use this idea.
- Anthropic's API reference documents messages.count_tokens as the supported method for Claude 3 and the Claude 4 family, accepting the same payload as messages.create and returning input_tokens.
- OpenAI's tiktoken library ships encoding_for_model() that returns cl100k_base for GPT-4 and o200k_base for GPT-4o, GPT-5, and o-series models.
- Google Gemini's billing page in 2026 lists character-based pricing for input and output, distinct from token-based pricing on OpenAI and Anthropic.
- HuggingFace transformers AutoTokenizer.from_pretrained loads the exact tokenizer that ships with Llama 3, Mistral, and Qwen 3.5 checkpoints, giving byte-exact counts for open-weight models.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour service makes 50 LLM calls per second to Claude. Can you afford to call count_tokens before every one?
QIf you have to estimate Claude token counts locally without count_tokens, what's the least-bad approach?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Reusing tiktoken cl100k_base or the deprecated Claude 2 tokenizer to estimate Claude 3 token counts, then trusting the number for billing or context-budget logic.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.