A finance leader hands you a spreadsheet ranking three LLM providers by published $ per million output tokens. Explain why the ranking is misleading even when the prices are accurate, and describe the comparison you would run instead. Cover the role of tokenizer family, the typical magnitude of the gap, and the unit that actually matters to a buyer.
Free · 2 AI evals / day
Different LLM providers use **different tokenizers**, and a tokenizer is what turns raw text into the discrete pieces (tokens) the model actually processes. OpenAI uses one BPE vocabulary (tiktoken with the cl100k or o200k family), Anthropic uses Claude's tokenizer, Meta's Llama-3 uses its own 128k-vocabulary BPE, Google's Gemini uses yet another. They were trained on different corpora with different merge rules, so the same sentence does not split into the same tokens across providers.
For English prose, the spread between major families is roughly **10 to 30 percent**. For code, non-English languages, or domain text, the gap can be larger; a tokenizer trained heavily on English will fragment Japanese or Arabic text far more than one trained multilingually. The practical effect is that 'a million output tokens' means a different amount of delivered text at each provider.
So when a finance spreadsheet ranks providers by **$/million output tokens**, it is comparing prices denominated in different units. If provider A is $0.50/M tokens and provider B is $0.55/M tokens, you cannot conclude A is cheaper without knowing how many tokens each one needs to produce the same response. If A's tokenizer is 20 percent more verbose than B's on your corpus, A's effective cost is $0.60/M of B-equivalent tokens, which is more expensive, not less.
The correct comparison fixes the unit of value at something tokenizer-independent: characters of input text, words of output, or a normalized reference corpus. Tokenize a representative sample through each provider's tokenizer, multiply by their per-token rate, and compare the total dollars. That number is honest. The headline $/M-token rate alone is not.
**Tokens are not a universal unit.** Each provider's tokenizer is trained on its own corpus with its own BPE merge rules and its own vocabulary size. The same paragraph splits into a different number of tokens at each provider, typically a 10 to 30 percent spread on English prose, and larger on code or non-English text.
That means $/M-token prices are denominated in **different units across providers** and only comparable within a family. A spreadsheet that ranks providers by $/M output tokens is comparing prices in different currencies as if the exchange rate were 1.00. The error compounds because the priced direction matters: input tokens are usually 3 to 5x cheaper than output, so output verbosity dominates the comparison. If provider A's tokenizer produces 20 percent more output tokens for the same delivered text, A's effective price is 20 percent higher than its headline rate.
The **honest comparison** fixes the unit of value at something tokenizer-independent. Take a representative corpus of input prompts and target outputs (real production traffic if available), tokenize that corpus through each provider's tokenizer to get per-provider token counts, multiply by each provider's $/token rate for input and output separately, and compare the total dollars per response or per million-character delivered.
This is exactly how serious procurement decisions actually work. The 'cheapest by headline rate' provider is often not the cheapest by delivered cost. The unit that actually matters to the buyer is dollars per useful answer; the tokenizer is just the intermediate accounting layer.
Quick reference
TL;DR
Each provider's tokenizer turns the same text into a different number of tokens. $/M-token rates compare only within a family; across providers, normalize to a shared source corpus.
Imagine three coffee shops all charging $5 per cup. Sounds equal, until you notice their cups hold different amounts: 8 oz, 12 oz, and 16 oz. The dollar per cup price tells you nothing about who is cheapest per ounce of coffee. LLM tokens work the same way. Each provider has its own cup size, that is what a tokenizer is, and the same paragraph of text needs more or fewer cups depending on whose cup you use. To compare honestly you have to convert back to ounces, which is to say, to the actual amount of text. Tokenize the same paragraph through each provider's tokenizer, count the cups you got, multiply by the per-cup price, and only then can you say who is cheaper.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Comparing different model sizes from the same provider (e.g., Claude Haiku vs Claude Sonnet) Same tokenizer within a provider family, so $/M tokens is directly comparable. The complication only arises across providers.
Quick back of envelope estimation for an internal hackathon If you do not need to be right within 30 percent, headline rates are fine as a first cut. The normalization matters when decisions are real and at scale.
Workloads dominated by cached input (heavy prompt caching, 90%+ hit rate) Cached input is so much cheaper that the tokenizer gap on cached tokens nearly disappears. Output gap still matters, but the total spread compresses.
Quick scan · 2–4 min
Everything important, quickly.
30 sec: tokens are vocabulary-specific, not a universal unit. 1 min: typical gap is 10-30 percent on English, larger on code and non-English. 1 min: output dominates because it is priced higher and is the verbose direction. 1 min: procurement workflow, tokenize same corpus through each tokenizer, multiply by rate, compare dollars.
Side-by-side
Comparison unit
Tokenizer-independent?
What it tells you
When to use
$/M tokens (headline)
No, family-specific
Provider's pricing within its own unit
Within-family rate changes only
$/M characters input + $/M characters output
Yes
Honest cross-provider total cost
Procurement decisions
$ per representative request
Yes (corpus-fixed)
Practical real-traffic cost
Pre-purchase load testing
$ per delivered useful answer
Yes (semantic)
What the buyer actually cares about
Production cost dashboards
Real-world examples
Real products, models, and research that use this idea.
1OpenAI's tiktoken library is published precisely so procurement teams can count tokens for their corpus before signing a contract.
2Anthropic publishes a count_tokens endpoint for the same purpose; treat 'we estimated' as a procurement red flag.
3On Japanese text, Llama-2 (older tokenizer) used roughly 2x more tokens per character than GPT-4 (cl100k), making nominally 'cheaper' Llama hosted endpoints more expensive on Japanese workloads.
4Anthropic's Claude tokenizer compresses JSON and code competitively with OpenAI's o200k; that matters for tool-use workloads where structured output dominates.
5When DeepSeek-V3 was released, third-party providers measured a tokenizer that was 15-20 percent more verbose than GPT-4 on English, which changed the effective cost comparison meaningfully.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat workloads see the largest tokenizer-driven cost variance across providers?
Sign in or go Pro to see the approach for this follow-up.
QAnthropic and OpenAI both offer prompt caching. Does that change the procurement math?
Sign in or go Pro to see the approach for this follow-up.
QIf two providers happen to use the exact same tokenizer (e.g., a Mistral-derivative and Mistral itself), is $/M tokens then directly comparable?
Sign in or go Pro to see the approach for this follow-up.
QWhy is the output-token gap usually more important than the input-token gap?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating $/M tokens as a universal price unit across providers. Tokens are vocabulary-specific; the same sentence is a different number of tokens at each provider. Always normalize to a tokenizer-independent unit before comparing.