Estimate the token cost difference for processing 1M Spanish/French product descriptions vs. an English-only baseline, and describe correct budget planning.
Your team's English-only embedding pipeline processes 1M product descriptions averaging 150 tokens each, costing $X at $0.10/1M tokens. You are expanding to Spanish and French. Describe how tokenizer fertility affects your cost estimate, quantify the expected token inflation, and outline the correct methodology for building an accurate budget before the pipeline launches.
Spanish and French tokenize about 1.1x to 1.5x more verbosely than English, so the same 1M descriptions cost 20% to 50% more, and the right budget comes from measuring fertility on real data, not a rule of thumb.
Imagine shipping the same gift in different boxes. English packs it in a small box, but Spanish and French need a bigger box for the same gift because of accents and longer word endings. You pay by box size, not by gift, so the foreign-language shipments cost more even though the contents match. To budget the move, you do not guess the box sizes; you actually pack a sample of real gifts in each language and weigh them, then multiply by how many you are sending.
Detailed answer & concept explanation~6 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: baseline 150M = $15 + fertility 1.1-1.5x + cost = tokens x price + 1.2x = $18, 1.5x = $22.50 + sample real data + p95 buffer + per-language rate limits + re-measure on encoding change.
import tiktoken
enc = tiktoken.encoding_for_model("text-embedding-3-small")
def budget(samples: list[str], total_docs: int, price_per_1m: float = 0.10):
counts = [len(enc.encode(d)) for d in samples]
mean = sum(counts) / len(counts)
p95 = sorted(counts)[int(0.95 * len(counts)) - 1]
est_tokens = mean * total_docs
est_cost = est_tokens / 1_000_000 * price_per_1m
return {"mean": mean, "p95": p95, "tokens": est_tokens, "cost": est_cost}
# Run per language on ~1000 real sampled descriptions each
es = budget(spanish_samples, 1_000_000)
fr = budget(french_samples, 1_000_000)| Scenario | Total tokens | Cost at $0.10/1M |
|---|---|---|
| English baseline (1.0x) | 150M | $15.00 |
| Spanish/French at 1.2x | 180M | $18.00 (+20%) |
| Spanish/French at 1.5x | 225M | $22.50 (+50%) |
Real products, models, and research that use this idea.
- OpenAI bills embeddings and GPT-5.5 calls per token, so a Spanish and French rollout raises the bill even at identical document counts.
- tiktoken is the tool teams use to measure per-language fertility on real product descriptions before committing a budget.
- Retrieval products like Perplexity feel the same tax at the generation stage, where high-fertility languages consume more of the priced context.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would the budget change if you migrated this pipeline from cl100k_base to o200k_base?
QHow do you size the p95 buffer for descriptions with extreme outliers?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Assuming the Spanish and French rollout costs the same as English because the document count is identical. Higher fertility means more tokens per document, so cost rises 20% to 50%.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.