Given a TPM rate limit, how do you compute supportable concurrent users for a feature?

Attempt it

Explain how to convert a tokens-per-minute (TPM) rate limit into supportable steady-state concurrent users for a feature. State the formula and discuss what knobs change the answer.

Free · 2 AI evals / day

Quick reference

TL;DR

Divide TPM by tokens per request for a sustainable request rate, then apply Little's law (concurrency equals arrival rate times response duration) to get supportable concurrent users.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a kitchen allowed to use only 200 ingredients per minute, total. Each dish needs about 5 ingredients, so the kitchen can start 40 dishes a minute. But a dish also sits on the counter while it finishes cooking. If each one takes 10 seconds to plate, several dishes are always mid-cook at any instant. Count how many are simultaneously in progress and that is how many customers you serve at once. Two things help. Smaller dishes use fewer ingredients each. Faster cooking frees the counter sooner. Smaller dishes win twice, because they use fewer ingredients AND finish sooner, so even more customers can be served at the same moment.

Key concepts

API rate limits Little's law Capacity planning Throughput vs concurrency

Detailed answer & concept explanation

~8 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

4 min: TPM to requests per minute, then Little's law to concurrency, a worked example, the output-length double lever, RPM as a separate ceiling, and tail-aware planning.

Real-world examples

Real products, models, and research that use this idea.

OpenAI publishes per-model TPM and RPM tiers; teams shipping on GPT-5.5 size concurrency from their org's tier limits before launch.
Anthropic's Claude Opus 4.7 API exposes separate input and output token-per-minute buckets, so the capacity math must split the two.
Together AI and Fireworks let teams buy dedicated capacity when shared-endpoint TPM tiers cap concurrent users too low.
Datadog and Helicone dashboards plot per-call token distributions and latency, the exact inputs this Little's law calculation needs.
Anthropic and OpenAI prompt caching cuts input tokens to roughly 10 percent, raising the request rate but leaving decode duration unchanged.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does halving output length raise concurrent-user capacity more than halving input length?

Sign in or go Pro to see the approach for this follow-up.

QWhen does the RPM limit bind before the TPM limit, and how do you detect it?

Sign in or go Pro to see the approach for this follow-up.

QHow does the calculation change for a streaming chat feature versus a batch summarization job?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Reporting requests per minute as the concurrent-user count. Throughput is a rate; concurrency is a level. You must multiply the rate by response duration via Little's law to bridge them.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.