Spot the flaws in this per tenant cost cap design
Click any words you think contain an error. Click again to unmark.
Three flaws: nightly reconciliation enforces too late, fleet averages hide per-call variance, and midnight auto-reenable plus unbounded streams reopens the overrun every day.
Imagine a parent gives a kid a monthly allowance and tells them to spend no more than 100 dollars. The parent only checks the receipts once a night and only ever notices yesterday's total. If the kid spent 500 dollars in one afternoon, the parent will not know until the next morning. Worse, the parent also estimates the kid's spending by counting how many trips to the store they took, not what they bought. A single trip to buy a TV gets the same estimate as a trip to buy gum. And every midnight the parent gives the kid the wallet back even though the allowance is already spent. The cap exists on paper, but in practice the kid spends whatever they want until each new day starts.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This v1 design is the most common shape teams ship for cost caps the first time, and it fails in exactly the same three ways every time. The bug is not that any one line is obviously wrong on its own; the bug is that enforcement is decoupled from the spend event along three axes at once: time (nightly), cardinality (request count vs tokens), and state (midnight reset plus unbounded streams). Each decoupling looks survivable in isolation. Together they make the cap purely cosmetic.
This walkthrough takes the three flaws one at a time, explains why each one breaks at production scale, and lays out what the correct shape looks like.
Mental model: a cap that runs after the spend is a report. A cap that runs before the spend is enforcement. Reports and enforcement are different products.
Flaw 1: temporal decoupling. Nightly is not enforcement
The bug. A nightly batch job that reads yesterday's billing export and disables tenants who exceeded their budget gives you a 24-hour window in which a tenant can spend any amount. The worst-case spend in that window is bounded only by your provider's rate limits and the tenant's wall-clock available time, not by the budget you set.
Why this happens. Reports are cheap to build, especially when the billing data is already exported nightly for finance. Teams reach for the cheap option, ship it, and call the box checked. The shape only fails when a single tenant has a runaway loop or a malicious user, and by definition that case is rare enough that v1 will never see it in testing.
The fix. Move enforcement to the request path. Each tenant has a ledger in low-latency storage (Redis is the standard choice). Before any model call, atomically debit an estimated cost against the ledger using a Lua script or a transactional UPDATE. After the model returns, settle the reservation with actuals. Nightly reconciliation does not go away; it moves to a different job, one that compares the ledger to the provider's truth and writes drift corrections. Reconciliation is the truth source. The ledger is the control plane.
Why atomicity matters. Even with a real-time ledger, a naive check then call design (read balance, compare, then debit) breaks under concurrency. Fifty parallel requests all read the same balance and all pass. The atomic reserve closes this hole. Lua scripts in Redis are atomic per-shard, and Postgres can do this with SELECT ... FOR UPDATE plus an UPDATE ... WHERE remaining >= cost.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LiteLLM Proxy explicitly warns against post-hoc enforcement and ships per-key real-time budgets in 2026.
- Helicone and Langfuse both expose per-request cost tracking specifically because fleet-average estimators are known to fail.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you keep the ledger from becoming a single point of failure?
Redis with replication and a Postgres source of truth for reconciliation. Fail open at low risk thresholds and fail closed near the cap. Per-tenant cached last-known balance in the worker process for the case where the ledger is briefly unreachable.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating cost caps as a reporting problem rather than an enforcement problem. Reports tell you what already happened; enforcement happens before the call lands at the provider.
60 second bullets to scan on the way to the call.
Why does nightly reconciliation fail as an enforcement mechanism?
What makes a fleet-wide average a bad cost estimator for LLM spend?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.