Fill in the libraries used to measure context size before sending a call
OpenAI uses tiktoken (local BPE library); Anthropic exposes messages.count_tokens on the SDK, which calls a server endpoint for the exact count.
Imagine mailing a heavy package: you weigh it on a scale to make sure the post office will accept it. For LLM calls, tokens are the weight and the API has a maximum. OpenAI gives you a kitchen scale you can keep on your counter, that is tiktoken, a small library that counts tokens locally and fast. Anthropic does not ship a portable kitchen scale; instead they let you call their official scale by phone before you send the package. That phone call is messages.count_tokens. Either way you weigh first, send second, and never get surprised by a rejected package at the counter.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every LLM workflow eventually hits the same wall: a request that looked fine in development gets rejected at runtime for exceeding the context limit, or a cost forecast based on character counts undershoots actual billing by a third. Both problems trace to the same root cause, not measuring tokens before sending.
The two big providers solved client-side token counting in opposite ways. Understanding the choice matters because it shapes how you build a budgeting layer, when you can count offline, and what corners you can cut safely.
OpenAI's path: ship the tokenizer
OpenAI publishes their tokenizer as a static BPE encoding. The tiktoken library is the official Python implementation; it ships the encoding tables and runs entirely on the client. There is no network call, no rate limit, no version skew between client and server because the encoding is the contract.
The two encodings you encounter most are cl100k_base (used by GPT-4, GPT-3.5-turbo, and embeddings models from that era) and o200k_base (used by GPT-4o, o-series reasoning models, and the GPT-5 family). The library exposes encoding_for_model(model_name) so you do not have to memorize the mapping; if you pass gpt-5 it returns the o200k_base encoding.
For chat completions there is a per-message overhead (a handful of tokens for role markers and message framing) that pure text encoding does not capture. The OpenAI cookbook documents the formula: a small constant plus the encoded length of each field. Budget-enforcement code that ignores this overhead undercounts by 3 to 8 tokens per message, which compounds in long conversations.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's official cookbook uses tiktoken for chat-message token accounting, including the per-message overhead tokens.
- Anthropic's SDK example for budgeting prompts calls client.messages.count_tokens with the exact request payload before sending.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour tiktoken count and Anthropic's count_tokens disagree for what should be the same payload. What could explain the gap?
Walk through tokenizer differences across families, the per-message overhead, system-prompt framing tokens, and tool-schema serialization.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Using the wrong encoding name for the model family or assuming string length in characters maps to token count, which under-counts non-ASCII text and code by a wide margin.
60 second bullets to scan on the way to the call.
Name the canonical client-side tokenizer for OpenAI
Name the canonical token-counting method for Anthropic
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.