Predict supportable concurrent user count under a 200k TPM rate limit
An API tier has a 200,000 tokens per minute (TPM) rate limit. A feature's typical request uses: - 4,000 input tokens - 1,000 output tokens (5,000 tokens total per request) - Average response duration (wall clock from request to last token): 10 seconds Compute, assuming steady state usage that fully utilizes the rate limit: 1. Sustainable requests per minute (rounded down to integer) 2. Sustainable concurrent users (assuming each active user blocks for the 10 second response duration) Report both numbers.
Divide the TPM budget by tokens per request to get 40 req/min, then apply Little's law (rate times duration) to get about 7 concurrent users.
Picture a toll booth that lets exactly 200,000 coins through per minute. Each car carries 5,000 coins, so at most 40 cars pass per minute. That is your request rate. Now ask a different question: how many cars are physically on the bridge at once? That depends on how long each car takes to cross. If a crossing takes 10 seconds, then in any single snapshot you only see the cars that started in the last 10 seconds. Forty cars per minute means roughly seven started in the last ten seconds, so about seven are on the bridge right now. Rate and occupancy are two separate things: the toll limit caps throughput, but crossing time decides how crowded the bridge looks at any instant.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This question looks like a one-line division, and the first half is. The real test is whether you separate two quantities that share the same inputs but answer different operational questions: how many requests you can sustain per minute, and how many run at the same instant.
Throughput is a rate, measured per unit time. Concurrency is an occupancy, a snapshot count with no time unit attached. They are linked, but the link is not equality. Confusing them is the single most common error on capacity-planning questions, and it leads directly to under-provisioned connection pools and mysterious queueing under load. A team that sizes a thread pool or a semaphore to 40 because the math said 40 requests per minute will starve under its own success, because the pool only ever needs to hold the seven that are actually in flight, and the other 33 are spread across the rest of the minute.
The bridge between the two is Little's law, one of the most useful results in queueing theory. It is distribution-free, which means it holds regardless of how arrivals or service times are distributed, as long as the system is stable. That generality is exactly why it shows up in capacity planning, warehouse logistics, hospital bed sizing, and LLM serving alike.
This deep dive walks the full calculation, derives the concurrency figure, explains why response duration rather than the token budget is the real concurrency lever, surfaces the throughput-versus-concurrency trap that interviewers set deliberately, and closes with the production caveats that separate a textbook answer from an operational one.
Step one: from a token budget to a request rate
A token-per-minute limit governs token consumption, not requests directly. To turn it into a request rate you need the token cost of one request, and that cost must sum both directions.
Each request here spends 4,000 input tokens and 1,000 output tokens. Both count against the budget, so the per-request cost is 5,000 tokens, not 1,000. Forgetting the input side is a frequent slip, especially because input dominates here at four-to-one.
The sustainable request rate is then the budget divided by the per-request cost:
This is a steady-state figure. It assumes you keep the pipe exactly full, with no idle gaps and no overshoot. In practice you run slightly under to leave error and retry headroom, but 40 per minute is the ceiling the limit permits.
Note how lopsided the request is: 4,000 of the 5,000 tokens are input. For a retrieval-augmented feature this is typical, since a long system prompt plus retrieved context dwarfs the user's question. The practical consequence is that the cheapest way to buy more throughput is often to shrink the prompt, not the completion. Prompt caching, where the provider charges a fraction of the normal rate for a repeated prefix, changes this arithmetic again, because cached input tokens may not count fully against the token budget. Always confirm how your provider bills cached prefixes before assuming the full 5,000 applies.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Quantity | Formula | Result here |
|---|---|---|
| Tokens per request | input + output | 4,000 + 1,000 = 5,000 |
| Sustainable req/min | TPM / tokens-per-request | 200,000 / 5,000 = 40 |
| Concurrent users | rate × (duration / 60) | 40 × (10/60) = 6.67 ≈ 7 |
Real products, models, and research that use this idea.
- OpenAI publishes per-model TPM and RPM tiers; capacity planning for a GPT-5.5 feature uses exactly this TPM-divided-by-tokens-per-request math.
- Anthropic's Claude Opus 4.7 API enforces both input and output token-per-minute limits, so engineers size pools against the binding constraint.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does halving response duration double supportable concurrency but not throughput?
Throughput is fixed by the token budget divided by tokens per request, which duration does not touch. Concurrency is rate times duration, so halving duration halves the per-request hold time and the budget then refills the freed slots with twice as many in-flight requests.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating requests per minute as the concurrency answer. The TPM limit caps throughput, but response duration is what sets how many requests are in flight at any instant.
60 second bullets to scan on the way to the call.
Why tokens per request must sum input and output before dividing
How a token per minute limit converts into a sustainable request rate
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.