Hosted LLM APIs throttle traffic on two independent axes: tokens and requests.
**TPM** stands for **Tokens Per Minute**. It caps the total number of tokens (prompt plus completion) that a single account or organization can submit in any rolling 60-second window. A request with a 50k-token prompt counts as 50k tokens against this limit, even if it is a single API call.
**RPM** stands for **Requests Per Minute**. It caps the number of API calls in the same 60-second window, regardless of how large each call is. One hundred tiny five-token requests counts the same as one massive 50k-token request.
Which limit you hit first depends entirely on your workload shape. A chat product making thousands of short turns per minute (10-200 tokens each) hits the RPM ceiling well before TPM. A document-summarization batch job processing a few 100k-token PDFs per minute hits TPM long before RPM.
Providers like OpenAI and Anthropic publish both numbers in their pricing tiers. As you spend more, both caps usually scale together, but they are tracked separately. Production systems that ignore one or the other end up surprised when traffic ramps and 429 errors arrive from an unexpected direction.
**TPM and RPM** are the two independent rate-limit axes that every hosted LLM API uses.
**TPM = Tokens Per Minute.** Caps the total tokens (input plus output) per 60-second rolling window. Workloads that send long prompts (RAG, document analysis, code-base review) or generate long outputs (creative writing, long-form summarization) push this ceiling first.
**RPM = Requests Per Minute.** Caps the count of API calls per 60-second window, independent of token volume. Workloads with many short calls (chat turns, classifier shots, tool-call dispatch, A/B testing) push this ceiling first.
A 100-token chat turn and a 50,000-token document analysis both consume 1 RPM unit, but the document consumes 500x the TPM. Conversely, 500 short chat turns burn 500 RPM units but might only burn 50,000 TPM, while a single long-context call burns 50,000 TPM but only 1 RPM.
Providers publish both: OpenAI's tier system (Tier 1-5) and Anthropic's organization-level rate-limit tiers each scale TPM and RPM together as account spend grows. Production engineering implications:
- Monitor both metrics independently with separate alerts.
- Use exponential backoff plus jitter on 429s, distinguishing TPM-limit retries (which need longer waits because tokens-consumed clears slowly) from RPM-limit retries.
- For RPM-pressured chat workloads, batch where possible. For TPM-pressured long-context workloads, use prefix caching (Anthropic 0.1x reads, OpenAI 0.5x reads) which reduces actual token billing but does NOT change the TPM counter on most providers (verify this per provider, it changes).
Quick reference
TL;DR
TPM caps total tokens per 60s window; RPM caps API calls per 60s window. Many small calls hit RPM first; few long-context calls hit TPM first.
Imagine a bakery with two limits: one says 'no more than 200 customers per hour' and another says 'no more than 1000 loaves served per hour'. A coffee shop with hundreds of tiny one-loaf orders hits the customer cap first. A wedding planner ordering 50 huge multi-loaf platters at once hits the loaf cap first. Hosted LLM providers do the same thing: RPM is the customer count, TPM is the loaf count, and which one blocks you first depends entirely on your workload shape.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Self-hosted deployments without external API throttling Capacity is governed by GPU memory and scheduler caps (max_num_batched_tokens, max_num_seqs), not by per-minute account quotas; the TPM/RPM framing does not transfer directly.
Per-user end-user facing rate limits in your own app You usually want a different abstraction (per-user requests, per-user spend) layered above the provider's account-level TPM/RPM; using provider quotas as the user-facing limit confuses scaling.
Batch endpoints with their own dedicated daily-quota throttle Batch APIs have separate higher TPM tiers and different RPM semantics (one batch submission can carry thousands of conceptual requests); the standard real-time rate-limit model does not apply.
Quick scan · 2–4 min
Everything important, quickly.
3 min: TPM as token volume cap + RPM as request count cap + workload patterns that hit each + retry and backoff per limit type + self-hosted analogues + tier scaling.
Side-by-side
Property
TPM
RPM
What it caps
Total tokens per 60s window
API calls per 60s window
Counts toward limit
prompt_tokens + completion_tokens
Each API call equals 1 unit
Workload that hits first
Long-context, RAG, document analysis
Chat turns, tool-call dispatch, classifier
Self-hosted analogue (vLLM)
max_num_batched_tokens
max_num_seqs
Real-world examples
Real products, models, and research that use this idea.
1OpenAI's Tier 1-5 system scales both TPM and RPM with account spend; Tier 5 grants on the order of millions of TPM for GPT-5.5.
2Anthropic publishes organization-level TPM and RPM for Claude Opus 4.7 and Sonnet 4.6, with separate caps for input tokens and output tokens.
3Gemini 3.1 Pro on Google AI Studio enforces RPM and TPM at the project level, with quota increase requests through the GCP console.
4Production teams running RAG pipelines on Claude Sonnet 4.6 hit TPM first because each retrieval injects 5-20K context tokens per query.
5vLLM deployments on Llama 4 Maverick expose `max_num_batched_tokens` and `max_num_seqs` as the self-hosted analogues to TPM and RPM.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you architect retries against a 429 response that does not specify which limit was hit?
Sign in or go Pro to see the approach for this follow-up.
QWhy might a single self-hosted vLLM cluster on H100s outperform a hosted provider's rate-limited tier for long-context workloads?
Sign in or go Pro to see the approach for this follow-up.
QWhat happens to the TPM counter when prefix caching gives you a 0.1x cache read?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating TPM and RPM as redundant. They throttle completely different axes; your workload pattern decides which one bites first.