Your multi-tenant product routes all LLM traffic through a shared gateway, and one tenant's batch scripts keep starving everyone else. Design per tenant rate limiting: decide what you meter, where you enforce it, and what a throttled tenant experiences.
Meter tokens and concurrency per tenant at the gateway. Reserve estimated input tokens, debit output tokens mid-stream. Return 429 with Retry-After. Keep a global guard under the provider's own limits.
Picture a buffet where one customer keeps loading a wheelbarrow while everyone else holds a plate. Counting trips through the line does not fix it because the wheelbarrow customer only goes once. You have to weigh the food on the way in. That is what tokens are for an LLM gateway: the right unit to count. Every customer gets a daily food budget by weight, and the door scale weighs the load before it goes onto the plate and again as more gets piled on. When a customer hits their weight cap, the door politely says "come back in twenty minutes" with a clear note about how full their bucket is. Above all customers, the kitchen has its own daily limit so no one customer plus a busy day can break the kitchen for everyone.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A shared LLM gateway is a small queueing-theory problem with an unusually unfair workload distribution. The cost of one request can vary by three orders of magnitude depending on prompt and completion length, so the usual web-service rate limit primitives (requests per second, requests per minute) measure the wrong thing. The job of per-tenant rate limiting is to keep one tenant's expensive workload from monopolizing capacity others paid for, while still being a polite, predictable surface for legitimate traffic to back off against.
This walkthrough breaks down the metering unit, the lifecycle of a request as it passes through the gate, the response format that lets well-behaved clients self-throttle, the degradation paths that turn a defensive primitive into a product feature, and the global guard above all tenants that keeps your upstream relationship healthy.
Mental model: the gate is a meter at the door (tokens, not feet) plus a velvet rope (concurrency) plus a soft override for VIPs (graceful degradation), plus a circuit breaker at the kitchen door (global guard).
Why tokens and concurrency are the right units
Tokens map to cost and GPU time
The cost of serving an LLM request is dominated by tokens: input tokens consume prefill compute and KV-cache memory, output tokens consume sequential decode steps. A 100k-token document summarization request is hundreds of times more expensive than a 500-token chat message, even though both count as one request.
If you meter only requests, a tenant who sends one giant request per minute looks polite. Meanwhile they pin the GPU for 30 seconds at a stretch and other tenants queue behind them. Meter tokens and the cost becomes visible at the gate.
Concurrency catches the streaming-tail case
Tokens-per-minute alone is gameable. A tenant could open 60 simultaneous low-rate streams and stay under a generous TPM budget while occupying every worker slot. Cap concurrent in-flight requests per tenant separately. Typical numbers: pricing tier T1 gets 4 concurrent, T2 gets 16, T3 gets 64.
Concurrency cap also helps with backpressure under upstream provider slowness: when one slow upstream call holds a slot for 60 seconds, the cap stops a tenant from accumulating dozens of pending slots.
What the tier sells
The pricing tier maps directly to two numbers: tokens per minute and concurrent requests. Optionally a daily or monthly token cap on top, for billing protection. Customer dashboards show these explicitly; nothing about throttling should be invisible to the tenant.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic and OpenAI both publish per-organization TPM and RPM limits and return Retry-After plus structured rate limit headers. Most enterprise gateways mirror this convention.
- AWS Bedrock applies per-account model-specific TPM and RPM limits; multi-tenant products on Bedrock implement their own per-tenant gate inside their account budget.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the token bucket handle a tenant who sends 100 requests simultaneously after a quiet period?
Burst capacity equals bucket size minus current usage. Size the bucket to the largest legitimate burst (often 5 to 10 minutes of steady-state TPM). Beyond that, requests get 429 even after a quiet idle. Tune per tier.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Metering requests per second only. A tenant sending one 200k-token prompt per second uses 200 times the GPU of a tenant sending 100-token prompts, but the RPS counter says they are equal.
60 second bullets to scan on the way to the call.
Why tokens beat requests as the metering unit
How input-token estimation works and where its error bands matter
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.