Estimate the per call cost of a typical RAG chatbot using GPT-4o-mini.
Input 1700 tokens at 0.15 USD per million plus output 250 tokens at 0.60 USD per million is roughly 0.0004 USD per call, about 1,200 USD per month at 100k calls per day.
Imagine paying for a taxi by two meters at once, one ticking on every kilometre you ride in (input) and a second ticking on every kilometre the driver speaks an answer (output). The output meter ticks four times faster per kilometre, but in this trip the input is much longer (about seven times the output) because the retrieved documents take up most of the ride. So even though the output rate is higher, the input still ends up costing more per trip. Add the two meters and one taxi ride costs a tiny fraction of a cent. Run a hundred thousand rides a day and the bill quietly climbs to about forty dollars.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
LLM cost arithmetic is a staple of system design interviews because it is concrete, verifiable, and reveals whether the candidate has actually run a production LLM workload. The numbers in this question are simple enough to do in your head with practice, and the structural lessons (input dominates RAG cost, retrieved context inflates the bill) generalise across every commercial API.
The interviewer is not looking for an exact dollar figure. They are looking for the candidate to decompose the call into input and output, recognise the asymmetric pricing, identify that retrieved context dominates input volume, project to daily and monthly cost at realistic traffic, and name at least one optimisation lever. Candidates who quote a single number without showing the work are signalling that they have only ever read pricing pages; candidates who walk the decomposition cleanly are signalling that they have actually owned a production LLM cost budget.
This deep dive walks the per call arithmetic, the scaling to realistic production volumes, the four levers for cost reduction, and where the cost-vs-quality tradeoffs live. By the end you should be able to ballpark any RAG cost in under a minute and name three optimisations that move the needle.
The per call arithmetic
All major commercial APIs price input and output separately. GPT-4o-mini at 2026 prices is 0.15 USD per million input tokens and 0.60 USD per million output tokens, a 4x asymmetry.
For this question's scenario, the input total is 200 (user prompt) + 1500 (retrieved context) = 1700 tokens. Per-call input cost is 1700 * 0.15 / 1,000,000, which gives 0.000255 USD.
Output is 250 tokens. Per-call output cost is 250 * 0.60 / 1,000,000, which gives 0.00015 USD.
Total per call: 0.000255 + 0.00015 = roughly 0.0004 USD. That's 4 hundredths of a cent.
Notice the structural point: even though output is priced 4x higher per token, input is 6.8x larger by volume, so input cost dominates by 1.7x. The cost ratio of any RAG call is set by the ratio of retrieved-context volume to response volume.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI's pricing page lists GPT-4o-mini at 0.15 USD input and 0.60 USD output per million tokens in 2026, with prompt caching cutting input prices roughly in half on cached blocks.
- Anthropic Claude Haiku 4 (the closest equivalent in capability) sits at similar price points with batch-API discounts of 50 percent.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat if context grows to 5,000 tokens (heavy reranker dropped)?
Input becomes 5,200 tokens at 0.15 per million = 0.00078 USD. Output stays 0.00015 USD. Per-call doubles to roughly 0.001 USD. At 100k calls per day, that's 100 USD per day, 2.5x the original.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Quoting one combined price for input and output. They are priced separately, input is cheaper per token, but RAG context inflates input volume far above output.
60 second bullets to scan on the way to the call.
Input versus output pricing asymmetry for GPT-4o-mini
How to compute per call cost from token counts
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.