Estimate the token cost difference for processing 1M Spanish/French product descriptions vs. an English-only baseline, and describe correct budget planning.
Short answer·Hard·4.0 · 0·~3 min·Relevant atOpenAISarvam
Your team's English-only embedding pipeline processes 1M product descriptions averaging 150 tokens each, costing $X at $0.10/1M tokens. You are expanding to Spanish and French. Describe how tokenizer fertility affects your cost estimate, quantify the expected token inflation, and outline the correct methodology for building an accurate budget before the pipeline launches.
Free · 2 AI evals / day
Start with the English baseline. One million descriptions at 150 tokens each is 150 million tokens. At $0.10 per million tokens, that is $15.
The key idea is **fertility**, the number of tokens a tokenizer produces per word. Spanish and French run higher than English on an English-trained tokenizer, because accented characters and long word endings do not have dedicated single tokens, so they split into more pieces. Empirically the inflation is about 1.1 to 1.5 times.
Since you pay per token, more tokens means more money for the same content. At 1.2 times fertility you get 180 million tokens, which is $18, a 20 percent increase. At 1.5 times you get 225 million tokens, which is $22.50, a 50 percent increase.
To budget correctly, do not trust a rule of thumb. Sample real Spanish and French descriptions, run them through the exact tokenizer your API uses, and measure the actual tokens per document. Then multiply and add a safety buffer.
The baseline is 1M descriptions times 150 tokens, or 150M tokens, which at $0.10 per 1M tokens is $15.
The variable that moves the bill is **fertility**, tokens per word. On an English-biased tokenizer, Spanish and French run roughly 1.1 to 1.5 times English, because accented characters and morphologically rich endings lack dedicated merges and fragment into subwords. Cost is linear in tokens.
$$\text{cost} = \text{tokens} \times \text{price-per-1k}$$
At 1.2x fertility you have 180M tokens for $18, a 20 percent rise. At 1.5x you have 225M tokens for $22.50, a 50 percent rise. And that compounds across embedding, generation, and reranking stages.
The right methodology is empirical, not a published multiplier. Sample about 1,000 Spanish and 1,000 French descriptions from your real dataset. Tokenize with the exact encoding the API uses, `tiktoken` for OpenAI or HF tokenizers for open-source models. Compute mean tokens per description per language, multiply by 1M, and add roughly a 15 percent buffer for outlier-long descriptions. Finally, set per-language tokens per minute thresholds so a high-fertility language does not silently trip rate limits.
Quick reference
TL;DR
Spanish and French tokenize about 1.1x to 1.5x more verbosely than English, so the same 1M descriptions cost 20% to 50% more, and the right budget comes from measuring fertility on real data, not a rule of thumb.
Imagine shipping the same gift in different boxes. English packs it in a small box, but Spanish and French need a bigger box for the same gift because of accents and longer word endings. You pay by box size, not by gift, so the foreign-language shipments cost more even though the contents match. To budget the move, you do not guess the box sizes; you actually pack a sample of real gifts in each language and weigh them, then multiply by how many you are sending.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A single-language English pipeline with stable inputs There is no cross-language fertility delta to model, so a per-language table is unnecessary overhead.
Committing a budget before sampling any real localized content Domain vocabulary shifts fertility, so a borrowed multiplier can leave you under-provisioned.
Reusing the budget after changing the model's tokenizer Per-language ratios differ between encodings, so the old table must be re-measured rather than scaled.
Quick scan · 2–4 min
Everything important, quickly.
4 min: baseline 150M = $15 + fertility 1.1-1.5x + cost = tokens x price + 1.2x = $18, 1.5x = $22.50 + sample real data + p95 buffer + per-language rate limits + re-measure on encoding change.
Code example
python
import tiktoken
enc = tiktoken.encoding_for_model("text-embedding-3-small")
def budget(samples: list[str], total_docs: int, price_per_1m: float = 0.10):
counts = [len(enc.encode(d)) for d in samples]
mean = sum(counts) / len(counts)
p95 = sorted(counts)[int(0.95 * len(counts)) - 1]
est_tokens = mean * total_docs
est_cost = est_tokens / 1_000_000 * price_per_1m
return {"mean": mean, "p95": p95, "tokens": est_tokens, "cost": est_cost}
# Run per language on ~1000 real sampled descriptions each
es = budget(spanish_samples, 1_000_000)
fr = budget(french_samples, 1_000_000)
Side-by-side
Scenario
Total tokens
Cost at $0.10/1M
English baseline (1.0x)
150M
$15.00
Spanish/French at 1.2x
180M
$18.00 (+20%)
Spanish/French at 1.5x
225M
$22.50 (+50%)
Real-world examples
Real products, models, and research that use this idea.
1OpenAI bills embeddings and GPT-5.5 calls per token, so a Spanish and French rollout raises the bill even at identical document counts.
2tiktoken is the tool teams use to measure per-language fertility on real product descriptions before committing a budget.
3Retrieval products like Perplexity feel the same tax at the generation stage, where high-fertility languages consume more of the priced context.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would the budget change if you migrated this pipeline from cl100k_base to o200k_base?
Sign in or go Pro to see the approach for this follow-up.
QHow do you size the p95 buffer for descriptions with extreme outliers?
Sign in or go Pro to see the approach for this follow-up.
QWhere does the fertility tax compound the most in a multi-stage pipeline?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming the Spanish and French rollout costs the same as English because the document count is identical. Higher fertility means more tokens per document, so cost rises 20% to 50%.