Same topic, related formats. Practice these next.
An API tier has a 200,000 tokens-per-minute (TPM) rate limit. A feature's typical request uses: - 4,000 input tokens - 1,000 output tokens (5,000 tokens total per request) - Average response duration (wall-clock from request to last token): 10 seconds Compute, assuming steady-state usage that fully utilizes the rate limit: 1. Sustainable requests per minute (rounded down to integer) 2. Sustainable concurrent users (assuming each active user blocks for the 10-second response duration) Report both numbers.
Divide the TPM budget by tokens per request to get 40 req/min, then apply Little's law (rate times duration) to get about 7 concurrent users.
Picture a toll booth that lets exactly 200,000 coins through per minute. Each car carries 5,000 coins, so at most 40 cars pass per minute. That is your request rate. Now ask a different question: how many cars are physically on the bridge at once? That depends on how long each car takes to cross. If a crossing takes 10 seconds, then in any single snapshot you only see the cars that started in the last 10 seconds. Forty cars per minute means roughly seven started in the last ten seconds, so about seven are on the bridge right now. Rate and occupancy are two separate things: the toll limit caps throughput, but crossing time decides how crowded the bridge looks at any instant.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
4 min: sum tokens per request, divide the TPM limit for the rate, state Little's law, apply rate times duration for concurrency, then discuss duration as the lever and the RPM-versus-TPM binding constraint.
| Quantity | Formula | Result here |
|---|---|---|
| Tokens per request | input + output | 4,000 + 1,000 = 5,000 |
| Sustainable req/min | TPM / tokens-per-request | 200,000 / 5,000 = 40 |
| Concurrent users | rate × (duration / 60) | 40 × (10/60) = 6.67 ≈ 7 |
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating requests per minute as the concurrency answer. The TPM limit caps throughput, but response duration is what sets how many requests are in flight at any instant.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.