Why cost per request becomes a first class SLO in LLMOps where it was never one in DevOps
Cost per request becomes a first-class SLO because a single LLM call can cost 100 to 10000 times a normal HTTP call and the cost varies wildly per request and per tenant.
Imagine your old website was a vending machine that charged a fraction of a cent every time someone pressed a button. Nobody bothered watching the change pile because each press was so cheap. Now imagine each button press costs anywhere from 5 cents to a full dollar, and the cost depends on how long the customer talks to the machine. Suddenly you watch every press. You put a counter next to each button, you alert when the counter spikes, and you ask why a single customer used five dollars of buttons in an afternoon. LLM calls are those expensive buttons, and cost per request is the counter that turns finance from a monthly surprise into a live dashboard.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A normal web service ships with three SLOs on the dashboard: latency, error rate, throughput. Operators stare at them all day because each one maps to user pain in real time. LLM-powered services keep all three and add a fourth: cost per request. The reason is structural rather than philosophical. The unit economics of an LLM call are unlike any previous compute primitive, and the resulting cost distribution behaves like a latency distribution: variable per request, regression-prone, operator-controllable, and capable of taking down the business.
This walkthrough explains why cost per request graduates to SLO status, what makes it different from generic cloud cost, how to instrument it, and the production discipline (CI gates, per-tenant budgets, alerting) that keeps it in bounds.
Mental model: in LLMOps, dollars per call is a latency-class metric. Track it like latency, alert like latency, gate PRs like latency. The finance team is the user.
Why cost per request behaves like latency, not like cloud spend
The unit-cost gap
A standard REST call on a well-tuned service costs a fraction of a cent: a few milliseconds of CPU plus a few KB of network. A single chat turn against a flagship LLM costs 5 cents to a full dollar. The gap is two to four orders of magnitude, and the upper end is not exotic; a 50k-token context window with an agent calling four tools easily lands at a dollar per trajectory.
Variable per request
LLM cost is a function of input tokens, output tokens, model choice, retries, tool calls, and cache hit rate. All of those are knobs that change call by call. A retrieval payload that grew from 2k to 5k tokens raises cost by 60 percent for every query touching it. A 5x retry pattern on a flaky provider triples cost without changing what the user sees.
Regression-prone
A prompt PR can raise cost permanently without a code deploy. A new few-shot example in the system prompt adds 200 tokens to every call. The eval CI catches a quality regression but does not catch the cost regression unless you wire one in. So the cost regression ships, and you find out at the end of the billing month.
Operator-controllable
Unlike traditional cloud spend (which is a function of traffic and provisioning), LLM cost is shaped by code and config changes made daily by engineers and content owners. That makes it an engineering metric, not a finance metric. Engineering metrics belong on the SLO dashboard.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Langfuse and LangSmith both compute `gen_ai.cost.usd` per span by joining token counts with a pricing table; the per-trace and per-tenant cost view is built in.
- Anthropic's Claude Opus 4.7 at $15 input and $75 output per million tokens makes a single agent trajectory with 50k context cost over a dollar without prompt caching.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you instrument cost per request end to end if you are starting from zero?
Add gen_ai.usage.input_tokens and gen_ai.usage.output_tokens to every LLM span via your OTel SDK or the framework auto-instrumentor. Maintain a pricing map (model id to input and output dollar per token). Compute cost in the observability backend, not at runtime, so pricing updates do not require a deploy.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Tracking total monthly bill instead of per-request cost. The bill is a lagging indicator; cost per request is the live signal that lets you catch a regression in minutes.
60 second bullets to scan on the way to the call.
The four LLM-service SLOs (latency, error rate, throughput, cost per request)
Why cost was not an SLO in pre-LLM DevOps
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.