Design per tenant cost caps that keep overruns under one percent
Your multi-tenant SaaS sells a prepaid monthly AI budget per tenant, and finance requires that no tenant overruns its budget by more than 1 percent. Design the metering and enforcement: how spend is tracked, how concurrent requests are handled, what happens mid stream, and what a capped tenant experiences.
Per-tenant ledger with atomic reserve then settle. Reserve estimated input plus max_tokens-bounded output before the call, settle actual usage after, and use the 1 percent as your error budget for drift.
Think of a prepaid debit card with a thousand dollars on it. Before the store swipes the card for your meal, the terminal puts a hold for the menu price plus a 20 percent tip. That hold is the reservation. Even if ten people swipe the same card at ten registers at the same minute, each register sees the remaining balance after the others reserved, so the card never goes negative. When you actually finish the meal, the real total replaces the hold. The 1 percent overshoot is just the rounding error from estimating the tip before you knew the exact bill. If you wait until end of month to add up receipts, the card has spent way past zero before anyone notices.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A 1 percent overrun SLA forces design decisions that no looser bound would. It rules out nightly reconciliation as enforcement, it rules out averaged cost estimation, it rules out checking the budget after the call. The only design that survives is a per-tenant ledger with atomic reserve then settle semantics, mid-stream output control, a soft-threshold ladder for user experience, and nightly reconciliation against provider truth. The hard part is not any single piece. The hard part is that all of them must work together for the 1 percent to hold at 50x concurrency, with token estimation drift, with settlement lag, and with mid-stream pricing variance.
This walkthrough lays out the design end to end: the ledger and its operations, where the 1 percent gets spent, how concurrency hardening works, what the user sees at each threshold, and how the system stays honest over time.
Mental model: treat the budget like a bank account with overdraft protection, not like a utility meter you read at month end.
The ledger and the reservation pattern
Storage. Each tenant has a row keyed (tenant_id, billing_period) with {budget_total, budget_remaining, reserved_total, last_reconciled_at}. Hot path lives in Redis for sub-millisecond latency; truth lives in Postgres and gets reconciled into Redis at startup and on every settlement.
The reserve operation is the atomic core. Before any model call, the caller computes an estimated cost and runs a Lua script (or a transactional UPDATE ... WHERE remaining >= cost RETURNING ...):
remaining = budget_remaining
IF remaining >= estimated_cost:
budget_remaining -= estimated_cost
reserved[request_id] = (estimated_cost, ttl=120s)
RETURN allowed
ELSE:
RETURN denied
The atomicity is what closes the concurrency hole. Fifty parallel requests all serialize through the script, each one sees the post prior debit balance, and the cap holds. Without atomicity, all fifty read the same balance and all pass.
The settle operation runs after the model call returns. It looks up the reservation, computes the difference between estimated and actual cost, and writes the adjustment back:
If the actual cost exceeded the reservation (rare, but possible if max_tokens was off or a tool-call expansion ran hotter than expected), the adjustment is negative and the ledger goes slightly more drawn down. Reservations carry a TTL so a crashed worker does not pin budget forever; the TTL is the max model wall-time plus a safety margin.
Why two steps and not one? A one-step debit (subtract actual cost after the call) is what produces the concurrency blow-out. Two steps let you accept the request only if the worst case fits, then refund the overestimate once you know the truth.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
-- Redis Lua: atomic reserve
local remaining = tonumber(redis.call('GET', KEYS[1]))
local cost = tonumber(ARGV[1])
local request_id = ARGV[2]
local ttl_seconds = tonumber(ARGV[3])
if remaining == nil or remaining < cost then
return {0, remaining or 0}
end
redis.call('DECRBY', KEYS[1], cost)
redis.call('SETEX', 'res:' .. request_id, ttl_seconds, cost)
return {1, remaining - cost}Real products, models, and research that use this idea.
- LiteLLM Proxy ships per-team and per-key budgets with pre-call cost estimation and post-call settlement.
- Helicone and Langfuse both expose tenant-level budget tracking and rate limits as first-class features in 2026.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you handle a tenant whose plan resets mid-month? Pro-rate the budget? Reset to zero?
Treat plan changes as a ledger event: write a credit or debit row with the proration calculation. Existing reservations stay valid. The next reconciliation picks up the new budget. Audit log every plan transition with the reasoning.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Checking the budget after the call instead of before. Concurrent requests each see headroom that does not exist and collectively blow the cap by far more than 1 percent.
60 second bullets to scan on the way to the call.
What is the concurrency hole in a naive check then call design?
Where does the 1 percent tolerance get spent across drift sources?
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.