Estimate the token cost difference for processing 1M Spanish/French product descriptions vs. an English only baseline, and describe correct budget planning.
Your team's English only embedding pipeline processes 1M product descriptions averaging 150 tokens each, costing $X at $0.10/1M tokens. You are expanding to Spanish and French. Describe how tokenizer fertility affects your cost estimate, quantify the expected token inflation, and outline the correct methodology for building an accurate budget before the pipeline launches.
Spanish and French tokenize about 1.1x to 1.5x more verbosely than English, so the same 1M descriptions cost 20% to 50% more, and the right budget comes from measuring fertility on real data, not a rule of thumb.
Imagine shipping the same gift in different boxes. English packs it in a small box, but Spanish and French need a bigger box for the same gift because of accents and longer word endings. You pay by box size, not by gift, so the foreign-language shipments cost more even though the contents match. To budget the move, you do not guess the box sizes; you actually pack a sample of real gifts in each language and weigh them, then multiply by how many you are sending.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A senior engineer is asked this question to see whether they can turn a fuzzy worry, foreign languages cost more, into a defensible number with a margin of error. The arithmetic is easy. The judgment is in knowing why a single multiplier is dangerous and what else breaks besides the dollar figure.
The core fact is that LLM APIs bill by the token, and tokenizers do not produce the same number of tokens for the same meaning across languages. An English-biased tokenizer fragments Spanish and French more, so a like for like catalog of product descriptions costs more in those languages even though the document count is identical.
We will do the baseline and inflated arithmetic, explain why you must measure on your own corpus rather than borrow a multiplier, lay out the sampling methodology with a safety buffer, and then cover the throughput failure mode that catches teams who budgeted only for cost.
Notice the shape of a strong answer before we start. It opens by anchoring to the known English number, introduces fertility as the one variable that moves it, gives a defensible range rather than a single figure, and then closes the loop with a measurement plan and the operational risks. An interviewer is listening for that structure as much as for the numbers, because it is the structure that survives when the specific prices and ratios change. The arithmetic is the easy part to teach; the discipline of bounding your uncertainty and naming the second-order failures is what marks the answer as senior.
The baseline and the inflated arithmetic
Begin with the number you control. One million product descriptions at 150 tokens each is 150 million tokens. At a price of $0.10 per million tokens, that is $15. This is the English baseline and the anchor for everything that follows.
Now introduce fertility, the tokens per word ratio. Spanish and French on an English-trained tokenizer run roughly 1.1 to 1.5 times English. Because cost is linear in tokens, the multiplier flows straight through. At 1.2 times you process 180 million tokens for $18, a 20 percent increase. At 1.5 times you process 225 million tokens for $22.50, a 50 percent increase.
Why state a range. Until you measure, the honest answer is a range, not a point. Lead with 10 to 50 percent and commit that you will tighten it with data. That signals you understand both the mechanism and its uncertainty.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
import tiktoken
enc = tiktoken.encoding_for_model("text-embedding-3-small")
def budget(samples: list[str], total_docs: int, price_per_1m: float = 0.10):
counts = [len(enc.encode(d)) for d in samples]
mean = sum(counts) / len(counts)
p95 = sorted(counts)[int(0.95 * len(counts)) - 1]
est_tokens = mean * total_docs
est_cost = est_tokens / 1_000_000 * price_per_1m
return {"mean": mean, "p95": p95, "tokens": est_tokens, "cost": est_cost}
# Run per language on ~1000 real sampled descriptions each
es = budget(spanish_samples, 1_000_000)
fr = budget(french_samples, 1_000_000)| Scenario | Total tokens | Cost at $0.10/1M |
|---|---|---|
| English baseline (1.0x) | 150M | $15.00 |
| Spanish/French at 1.2x | 180M | $18.00 (+20%) |
| Spanish/French at 1.5x | 225M | $22.50 (+50%) |
Real products, models, and research that use this idea.
- OpenAI bills embeddings and GPT-5.5 calls per token, so a Spanish and French rollout raises the bill even at identical document counts.
- tiktoken is the tool teams use to measure per-language fertility on real product descriptions before committing a budget.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would the budget change if you migrated this pipeline from cl100k_base to o200k_base?
Re-measure fertility per language under the new encoding; CJK gains most but European ratios shift too, so do not just scale the old numbers.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming the Spanish and French rollout costs the same as English because the document count is identical. Higher fertility means more tokens per document, so cost rises 20% to 50%.
60 second bullets to scan on the way to the call.
The baseline token and cost arithmetic
Why fertility is the variable that moves the bill
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.