Given a TPM rate limit, how do you compute supportable concurrent users for a feature?

API providers cap you on **tokens per minute** (TPM), and separately on requests per minute. The product question is how many people your feature can serve at the same time without hitting that ceiling. Start with the token budget. Every call to your feature spends roughly the same number of tokens: the input prompt plus the generated output. Divide the TPM ceiling by that per-call token count and you get how many requests per minute you can sustain. But requests per minute is a rate, not a count of people online at once. To get concurrency you need one more fact: how long each request takes. A request that runs for ten seconds occupies a slot for ten seconds. Little's law bridges the two. It says the average number of in-flight requests equals the arrival rate times the average time each request stays in the system. So concurrent users equals requests per second times response duration in seconds. The biggest lever is output length. Shorter outputs cost fewer tokens AND finish faster, so they raise capacity on both sides of the calculation at once.

**Rate limits** are token budgets, but products care about concurrent users. Bridging the two takes two steps. Step one converts the token ceiling to a request rate. Sustainable requests per minute equals TPM divided by tokens per request. Tokens per request is the typical input plus output for that feature's call. So 200k TPM at 5k tokens per call gives 40 requests per minute. Step two converts that rate to a concurrency level using Little's law: $$N = \lambda \cdot D = \frac{\text{TPM}}{60 \cdot \text{tok/req}} \cdot D$$ N is concurrent users. Lambda is arrival rate per second. D is mean response duration in seconds. At 40 requests per minute and a 10 second response, that is roughly 6.7 concurrent users. Three knobs move the answer. Output length is highest leverage because it appears twice. It sits in the denominator of step one and inside D, since longer generations take longer to decode. Decode speed in tokens per second sets D on its own. Prompt caching cuts input tokens and raises the request rate, but does not shrink D. Finally, TPM is not the only ceiling. RPM is a separate limit that can bind first on short calls. Most providers also allow brief bursts above steady-state TPM, so a sustained estimate is conservative for spiky traffic.