Predict supportable concurrent-user count under a 200k TPM rate limit
An API tier has a 200,000 tokens-per-minute (TPM) rate limit. A feature's typical request uses: - 4,000 input tokens - 1,000 output tokens (5,000 tokens total per request) - Average response duration (wall-clock from request to last token): 10 seconds Compute, assuming steady-state usage that fully utilizes the rate limit: 1. Sustainable requests per minute (rounded down to integer) 2. Sustainable concurrent users (assuming each active user blocks for the 10-second response duration) Report both numbers.
Divide the TPM budget by tokens per request to get 40 req/min, then apply Little's law (rate times duration) to get about 7 concurrent users.
Picture a toll booth that lets exactly 200,000 coins through per minute. Each car carries 5,000 coins, so at most 40 cars pass per minute. That is your request rate. Now ask a different question: how many cars are physically on the bridge at once? That depends on how long each car takes to cross. If a crossing takes 10 seconds, then in any single snapshot you only see the cars that started in the last 10 seconds. Forty cars per minute means roughly seven started in the last ten seconds, so about seven are on the bridge right now. Rate and occupancy are two separate things: the toll limit caps throughput, but crossing time decides how crowded the bridge looks at any instant.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: sum tokens per request, divide the TPM limit for the rate, state Little's law, apply rate times duration for concurrency, then discuss duration as the lever and the RPM-versus-TPM binding constraint.
| Quantity | Formula | Result here |
|---|---|---|
| Tokens per request | input + output | 4,000 + 1,000 = 5,000 |
| Sustainable req/min | TPM / tokens-per-request | 200,000 / 5,000 = 40 |
| Concurrent users | rate × (duration / 60) | 40 × (10/60) = 6.67 ≈ 7 |
Real products, models, and research that use this idea.
- OpenAI publishes per-model TPM and RPM tiers; capacity planning for a GPT-5.5 feature uses exactly this TPM-divided-by-tokens-per-request math.
- Anthropic's Claude Opus 4.7 API enforces both input and output token-per-minute limits, so engineers size pools against the binding constraint.
- vLLM and SGLang dashboards expose in-flight request gauges that track the Little's-law concurrency estimate against actual GPU occupancy.
- Together AI and Modal capacity calculators ask for tokens per request and average latency to project sustainable concurrent sessions.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does halving response duration double supportable concurrency but not throughput?
QWhen does the requests-per-minute limit bind before the token-per-minute limit?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating requests per minute as the concurrency answer. The TPM limit caps throughput, but response duration is what sets how many requests are in flight at any instant.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.