Your multi-tenant product routes all LLM traffic through a shared gateway, and one tenant's batch scripts keep starving everyone else. Design per-tenant rate limiting: decide what you meter, where you enforce it, and what a throttled tenant experiences.
Free · 2 AI evals / day
A multi-tenant LLM gateway is shared GPU time. If one tenant runs a batch job that floods the gateway, every other tenant slows down or starts seeing errors. **Per-tenant rate limiting** is the gate that prevents one customer from starving the rest.
The right unit to limit is **tokens**, not requests. A request that asks for a 50k-token document summary costs hundreds of times more GPU time than a chat message with 100 tokens. Counting requests alone makes a tenant who sends one huge job per minute look polite.
The pattern: assign each tenant a tokens per minute budget based on their pricing tier. Run a token bucket per tenant. When a request arrives, estimate the input tokens, check if the bucket has room, and if not, return a 429 status with a Retry-After header telling the client when to try again. As the response streams back, debit the output tokens too. That way a tenant who asks for a 30k-token generation cannot sneak under the gate by having a small prompt.
Well-behaved SDKs see the 429 and back off automatically. Less-well-behaved clients still hit the gate, but the gate holds. Higher-tier tenants can be offered a softer option: queue the work for later, or fall back to a cheaper model, instead of a hard rejection.
The gateway meters two things per tenant: **tokens per minute** (or per second, depending on how spiky the workload is) and **concurrent in-flight requests**. Tokens cover cost and GPU time; concurrency covers the long-generation case where a single request occupies a slot for tens of seconds.
**Enforcement runs at the gateway, before the upstream provider call.** The flow:
1. Tokenize the request locally (or use a cheap estimator) to get an input-token count.
2. Reserve that estimate against the tenant's token bucket. If the bucket is empty, return 429 with Retry-After and a JSON body describing the limit, the current usage, and the reset window.
3. Make the upstream call and stream the response.
4. As output tokens arrive, debit the bucket incrementally. If the request runs long and the bucket would go negative, you can either cut the stream (hard) or let it finish but block the next request (soft).
5. On completion, reconcile the reservation against the actual input and output counts.
**Higher tiers get graceful degradation** instead of hard rejection: a low-priority queue with a longer SLA, automatic fallback to a smaller model, or batched delivery overnight.
**Above all tenants, a global budget guard** mirrors the upstream provider's own rate limits. If the gateway aggregates close to the provider's TPM ceiling, throttle the lowest tiers first to keep the highest tiers responsive.
**Observability is part of the design.** Per-tenant usage dashboards, per-tenant 429 rates, time to reset histograms. Support needs to explain throttling, not discover it from a ticket.
Quick reference
TL;DR
Meter tokens and concurrency per tenant at the gateway. Reserve estimated input tokens, debit output tokens mid-stream. Return 429 with Retry-After. Keep a global guard under the provider's own limits.
Picture a buffet where one customer keeps loading a wheelbarrow while everyone else holds a plate. Counting trips through the line does not fix it because the wheelbarrow customer only goes once. You have to weigh the food on the way in. That is what tokens are for an LLM gateway: the right unit to count. Every customer gets a daily food budget by weight, and the door scale weighs the load before it goes onto the plate and again as more gets piled on. When a customer hits their weight cap, the door politely says "come back in twenty minutes" with a clear note about how full their bucket is. Above all customers, the kitchen has its own daily limit so no one customer plus a busy day can break the kitchen for everyone.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Single-tenant deployment with no shared GPU pool Per-tenant fairness is moot when there is only one tenant; a simpler global cap against the upstream provider suffices.
Workload is small and stays well under provider limits Building token buckets, reservations, and dashboards is real engineering; for a low-traffic product, a static client-side throttle is cheaper.
All traffic is batch with no latency expectations A queue with global concurrency cap handles fairness without needing per-tenant token buckets and 429 semantics.
Cost attribution can be done after the fact via logs If the goal is billing rather than protection, post-hoc metering on logs is simpler than real-time gating, though you lose the starvation defense.
Quick scan · 2–4 min
Everything important, quickly.
8 min: why tokens (and concurrency) beat requests, the reserve debit settle lifecycle at the gateway, 429 semantics and headers, tiered degradation paths, the global guard that mirrors the upstream provider's limits, observability.
Real-world examples
Real products, models, and research that use this idea.
1Anthropic and OpenAI both publish per-organization TPM and RPM limits and return Retry-After plus structured rate limit headers. Most enterprise gateways mirror this convention.
2AWS Bedrock applies per-account model-specific TPM and RPM limits; multi-tenant products on Bedrock implement their own per-tenant gate inside their account budget.
3LiteLLM and Helicone both ship per virtual key budgets that meter tokens (input plus output), enforce caps at the proxy, and surface usage in dashboards.
4Stripe, Plaid, and other high-volume APIs are the canonical reference for the rate limit header convention (X-RateLimit-*, Retry-After) that LLM gateways follow.
5Cloudflare AI Gateway exposes per-application token budgets with caching plus rate limits, demonstrating the gateway as product pattern.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does the token bucket handle a tenant who sends 100 requests simultaneously after a quiet period?
Sign in or go Pro to see the approach for this follow-up.
QWhat happens when input-token estimation is off by 30 percent on a single request?
Sign in or go Pro to see the approach for this follow-up.
QHow do you migrate a tenant from one tier to another without a deploy?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Metering requests per second only. A tenant sending one 200k-token prompt per second uses 200 times the GPU of a tenant sending 100-token prompts, but the RPS counter says they are equal.