For OpenAI models, the canonical client-side tokenizer is . For Anthropic models, the official approach is to call on the SDK to get an authoritative token count before sending.
Every LLM API charges by tokens, not by characters, and every model has a hard token limit. Before you send a request you want to know how many tokens it will cost. Two ecosystems, two tools.
**OpenAI**: use `tiktoken`. It is OpenAI's open-source BPE tokenizer library, runs locally, no network call, fast. Pick the encoding that matches your model: `o200k_base` for GPT-4o and GPT-5 family, `cl100k_base` for older GPT-4 and GPT-3.5. The library exposes helpers like `encoding_for_model('gpt-5')` so you do not have to memorize encoding names.
**Anthropic**: use the SDK method `client.messages.count_tokens(...)`. Anthropic does not ship a portable tokenizer because their tokenizer changes per model and is not published as a static encoding. Instead they expose a dedicated count-tokens endpoint that mirrors the messages-create payload shape. You pass the same `messages`, `system`, and `tools` arguments you would send to a real call, and get back a `{input_tokens: N}` response.
In both cases the workflow is: count first, then decide whether to send, trim, or split.
**OpenAI: `tiktoken`. Anthropic: `client.messages.count_tokens(...)`.**
The asymmetry reflects a design difference. OpenAI publishes their tokenizer as a static BPE encoding (`cl100k_base`, `o200k_base`) so a client library can replicate token counts offline. That makes `tiktoken` cheap, fast, and deterministic; you can budget inside a loop without paying network round-trips.
Anthropic treats tokenization as a model version specific artifact and does not publish a portable encoding. The trade is that you must call their endpoint to get an authoritative count, but the count you get is guaranteed exact for the model version you target. The SDK method takes the full message-create shape (system prompt, messages, tools, tool_choice) and returns input_tokens, which matters because tool definitions and image content participate in the count in non-obvious ways.
Other providers follow one of these two patterns. Google's Gemini SDK exposes `model.count_tokens(...)` (server-side, like Anthropic). Cohere ships a local tokenizer. Mistral and DeepSeek publish their tokenizers on Hugging Face Hub via `AutoTokenizer.from_pretrained(...)`. The principle is the same: count before sending, especially when context is non-trivial or when you are budgeting against a soft limit well below the hard one.
Quick reference
TL;DR
OpenAI uses tiktoken (local BPE library); Anthropic exposes messages.count_tokens on the SDK, which calls a server endpoint for the exact count.
Imagine mailing a heavy package: you weigh it on a scale to make sure the post office will accept it. For LLM calls, tokens are the weight and the API has a maximum. OpenAI gives you a kitchen scale you can keep on your counter, that is tiktoken, a small library that counts tokens locally and fast. Anthropic does not ship a portable kitchen scale; instead they let you call their official scale by phone before you send the package. That phone call is messages.count_tokens. Either way you weigh first, send second, and never get surprised by a rejected package at the counter.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You are doing a rough back of envelope sizing during a design review, not enforcing a budget at runtime A word-count heuristic is plenty for sizing; the SDKs are overkill at that altitude
You are running on a model that publishes its tokenizer on Hugging Face (Mistral, DeepSeek, Llama) AutoTokenizer.from_pretrained gives you a portable local count; reaching for a vendor-specific method is unnecessary
You only count tokens for billing reconciliation, not for pre-flight budgeting Use the usage field on the response itself; it is the ground truth for billing
Quick scan · 2–4 min
Everything important, quickly.
6 min: tiktoken versus messages.count_tokens, why the asymmetry exists, character heuristics and where they fail, tool schemas and image tokens, cache the prefix pattern, cross-provider abstraction.
Real-world examples
Real products, models, and research that use this idea.
1OpenAI's official cookbook uses tiktoken for chat-message token accounting, including the per-message overhead tokens.
2Anthropic's SDK example for budgeting prompts calls client.messages.count_tokens with the exact request payload before sending.
3Google Gemini's SDK exposes model.count_tokens, returning total_tokens for the full payload including inline images.
4LangChain's prompt-length helpers wrap tiktoken for OpenAI and the SDK method for Anthropic, so application code stays provider-agnostic.
5LlamaIndex's TokenCounter callback hooks into the active model's tokenizer to track input and output tokens per request.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QYour tiktoken count and Anthropic's count_tokens disagree for what should be the same payload. What could explain the gap?
Sign in or go Pro to see the approach for this follow-up.
QHow would you implement a budgeting layer that supports OpenAI, Anthropic, and Gemini with the same interface?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Using the wrong encoding name for the model family or assuming string length in characters maps to token count, which under-counts non-ASCII text and code by a wide margin.