Diagnose the reasoning error in 'a bigger tokenizer vocabulary always cuts our token bill'

Spot the error·Medium·4.0 · 0·~2 min·Asked atCitadel Elastic Oracle·Relevant atOpenAI

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

**Tokens are units of measurement, not units of money.** Every LLM API bills in tokens, but the price per token depends on the model. Two different models, even from the same provider, often have very different per-token prices. The argument in the question observes that the new model uses 30 percent fewer tokens to encode the same text. That is a real and useful property of a richer tokenizer. But the conclusion that the bill drops by 30 percent assumes the price per token is identical between the two models. It almost never is. The correct comparison is dollars per unit of text, not tokens per unit of text. Take a sample of your real workload, count the tokens each model produces for it, multiply by each model's listed per-token price for both input and output, and compare totals. A model with a richer vocabulary often charges more per token, because each token now carries more information; the price can erase the token-count win entirely or even reverse it. Fewer tokens does help one thing for sure: latency. Fewer tokens means fewer decode steps, which lowers time-to-first-token and time-per-output-token. That is a real win, but it is a latency win, not necessarily a dollar win.

**The argument confuses tokens with dollars.** Cost per request is tokens-per-request multiplied by price-per-token. The proposal holds the second factor implicitly constant, which is the bug. Different models have different per-token prices and the price difference can easily dominate a 30 percent token-count reduction. The correct metric is dollars-per-byte-of-text, or dollars-per-character if you prefer, evaluated on the actual corpus. Write it out: `dollars/byte = (tokens/byte) * (price/token)`. The argument has measured the first factor (tokens per byte, which dropped 30%) but assumed the second factor held constant. It rarely does. Real cases pull in both directions. Models with richer tokenizers often charge a premium per token, because each token is worth more information and the provider passes the value through. Sometimes the premium fully cancels the token-count saving, sometimes it overshoots and the richer-tokenizer model is actually more expensive per byte of text. The only way to know is to reprice on the real price sheet against a real workload sample. There is one win that the bigger tokenizer delivers for free: fewer decode steps. TPOT is per token, so a 30 percent reduction in token count cuts decode wall-clock by roughly 30 percent. TTFT is partly prefill-time over input tokens, which also benefits. These are latency wins worth real money on user-facing products, but they are not the same as the dollar saving on the per-token bill. The right framing for a migration decision: compute cost-per-byte and latency-per-byte on the real workload for both models, weight them by your product's value-of-latency function, and pick the model whose composite score wins. Skipping that arithmetic and trusting the headline token-count reduction is how teams ship a 'cheaper' model that turns out to be more expensive.

TL;DR

Fewer tokens does not mean fewer dollars. Cost = tokens × price-per-token, and different models have different per-token prices.

Memory aid

Easy to grasp

Imagine two grocery stores selling the same fruit but in differently sized boxes. One store packs three apples per box, the other packs four. The four-apple box has fewer boxes per pound of apples, but each box might also cost more. You cannot tell which store is cheaper just by counting boxes; you have to multiply boxes by box price to get dollars. Tokens are the boxes. The new model packs more text per token, so you have fewer tokens, but each token may also cost more. Always do the multiplication on the real price sheet before celebrating the box count.

Key concepts

Tokenization Inference optimization Pricing models Cost modeling

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Tokenizer choices are often discussed as if they were a direct cost lever, and they are not. A tokenizer determines how many tokens a piece of text becomes, which is one factor in the bill, but the price per token is a separate factor set by the provider's pricing decisions. The argument in the question slides those two factors together, takes a measurement of the first, and projects it onto the bill as if the second held constant. It almost never does.

The scenario is concrete: a corpus measurement shows the new model's tokenizer encodes the same text in thirty percent fewer tokens. The author then concludes the bill drops thirty percent. That conclusion is correct only if the per-token price on the new model is exactly equal to the per-token price on the old model, which is rarely true for any pair of real models worth comparing. The price asymmetry between the two models can erase the token-count saving, amplify it, or even reverse it.

This walkthrough builds the correct cost model, applies it to representative cases (GPT-4o vs GPT-3.5, Claude vs GPT, DeepSeek vs Llama), and explains the latency dimension that does benefit from fewer tokens regardless of the price sheet. By the end you should be able to evaluate any tokenizer-driven migration claim by reaching for the dollars-per-byte metric and decomposing input and output separately.

The cost formula and what the argument silently held constant

The bill for one API call is straightforward: number of input tokens times input price-per-token, plus number of output tokens times output price-per-token. For a workload at scale, sum across requests. There are exactly two factors per direction: a count and a price.

The corpus measurement in the question tells us something useful and concrete about the first factor: the new tokenizer produces thirty percent fewer tokens for the same text. The argument then makes its only error in a single step: it multiplies that token-count change by a price that it never actually checked, assuming the price held constant. Strip that assumption out and the argument has nothing to say about dollars.

The correct metric for a model-migration decision is cost-per-unit-of-text, typically expressed as dollars-per-million-words or dollars-per-thousand-characters or dollars-per-byte. It equals tokens-per-unit-of-text multiplied by price-per-token. Both factors must be measured. The first factor is what the corpus study delivers. The second factor is what the price sheet delivers. Neither factor alone tells you the cost direction.

This sounds elementary, and it is. The reason it routinely catches engineers is that the token-count number is concrete, locally measurable, and feels like a 'real' input to the cost. The price ratio is on a web page somewhere, often varies by SKU, tier, region, and date, and is easy to skip past. The bug pattern is to anchor on the measurement you ran yourself and to forget the one you would have to look up.

Why richer tokenizers tend to cost more per token

Input versus output and the decomposition that matters

The latency dimension and when it dominates the decision

How to actually run the migration analysis

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

GPT-4o uses the o200k_base tokenizer (200k vocab) vs cl100k_base on GPT-3.5/4, encoding the same text in ~10-30% fewer tokens on English but with a higher per-token output price.
Anthropic's tokenizer is similar in size to OpenAI's older models, so a Claude-to-GPT-4o migration changes both the tokenizer ratio and the price simultaneously; the net cost depends on language and direction.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy do providers tend to charge more per token on models with richer tokenizers?

Each token in a richer vocabulary carries more information and corresponds to more underlying serving cost (similar attention work over a sequence, larger embedding table). Providers price for value-per-token, so richer tokens command a premium. The net effect on cost-per-byte often nets close to flat, intentionally.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Equating fewer tokens with lower cost without checking that the per-token price held constant. A new model with a richer vocabulary typically prices each token differently; the comparison must be dollars per unit text, not tokens per unit text.

60 second bullets to scan on the way to the call.

How to write the cost formula explicitly: tokens per request times price per token
Why dollars per byte of text is the migration-decision metric, not tokens per byte

Primary sources. Browse if you want the original framing.