TPM and RPM each throttle a different axis, which one caps which?

Match pairs·Easy·4.0 · 0·~2 min·Relevant atOpenAI

Attempt it

TPM

RPM

Many small calls

Few long-context calls

Hosted LLM APIs throttle traffic on two independent axes: tokens and requests. **TPM** stands for **Tokens Per Minute**. It caps the total number of tokens (prompt plus completion) that a single account or organization can submit in any rolling 60-second window. A request with a 50k-token prompt counts as 50k tokens against this limit, even if it is a single API call. **RPM** stands for **Requests Per Minute**. It caps the number of API calls in the same 60-second window, regardless of how large each call is. One hundred tiny five-token requests counts the same as one massive 50k-token request. Which limit you hit first depends entirely on your workload shape. A chat product making thousands of short turns per minute (10-200 tokens each) hits the RPM ceiling well before TPM. A document-summarization batch job processing a few 100k-token PDFs per minute hits TPM long before RPM. Providers like OpenAI and Anthropic publish both numbers in their pricing tiers. As you spend more, both caps usually scale together, but they are tracked separately. Production systems that ignore one or the other end up surprised when traffic ramps and 429 errors arrive from an unexpected direction.

**TPM and RPM** are the two independent rate-limit axes that every hosted LLM API uses. **TPM = Tokens Per Minute.** Caps the total tokens (input plus output) per 60-second rolling window. Workloads that send long prompts (RAG, document analysis, code-base review) or generate long outputs (creative writing, long-form summarization) push this ceiling first. **RPM = Requests Per Minute.** Caps the count of API calls per 60-second window, independent of token volume. Workloads with many short calls (chat turns, classifier shots, tool-call dispatch, A/B testing) push this ceiling first. A 100-token chat turn and a 50,000-token document analysis both consume 1 RPM unit, but the document consumes 500x the TPM. Conversely, 500 short chat turns burn 500 RPM units but might only burn 50,000 TPM, while a single long-context call burns 50,000 TPM but only 1 RPM. Providers publish both: OpenAI's tier system (Tier 1-5) and Anthropic's organization-level rate-limit tiers each scale TPM and RPM together as account spend grows. Production engineering implications: - Monitor both metrics independently with separate alerts. - Use exponential backoff plus jitter on 429s, distinguishing TPM-limit retries (which need longer waits because tokens-consumed clears slowly) from RPM-limit retries. - For RPM-pressured chat workloads, batch where possible. For TPM-pressured long-context workloads, use prefix caching (Anthropic 0.1x reads, OpenAI 0.5x reads) which reduces actual token billing but does NOT change the TPM counter on most providers (verify this per provider, it changes).

Quick reference

TL;DR

TPM caps total tokens per 60s window; RPM caps API calls per 60s window. Many small calls hit RPM first; few long-context calls hit TPM first.

Memory aid

Easy to grasp

Imagine a bakery with two limits: one says 'no more than 200 customers per hour' and another says 'no more than 1000 loaves served per hour'. A coffee shop with hundreds of tiny one-loaf orders hits the customer cap first. A wedding planner ordering 50 huge multi-loaf platters at once hits the loaf cap first. Hosted LLM providers do the same thing: RPM is the customer count, TPM is the loaf count, and which one blocks you first depends entirely on your workload shape.

Key concepts

API rate limiting Tokens per minute Requests per minute Pricing and quotas Serving

Detailed answer & concept explanation

~4 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3 min: TPM as token volume cap + RPM as request count cap + workload patterns that hit each + retry and backoff per limit type + self-hosted analogues + tier scaling.

Side-by-side

Property	TPM	RPM
What it caps	Total tokens per 60s window	API calls per 60s window
Counts toward limit	prompt_tokens + completion_tokens	Each API call equals 1 unit
Workload that hits first	Long-context, RAG, document analysis	Chat turns, tool-call dispatch, classifier
Self-hosted analogue (vLLM)	max_num_batched_tokens	max_num_seqs

Real-world examples

Real products, models, and research that use this idea.

OpenAI's Tier 1-5 system scales both TPM and RPM with account spend; Tier 5 grants on the order of millions of TPM for GPT-5.5.
Anthropic publishes organization-level TPM and RPM for Claude Opus 4.7 and Sonnet 4.6, with separate caps for input tokens and output tokens.
Gemini 3.1 Pro on Google AI Studio enforces RPM and TPM at the project level, with quota increase requests through the GCP console.
Production teams running RAG pipelines on Claude Sonnet 4.6 hit TPM first because each retrieval injects 5-20K context tokens per query.
vLLM deployments on Llama 4 Maverick expose `max_num_batched_tokens` and `max_num_seqs` as the self-hosted analogues to TPM and RPM.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you architect retries against a 429 response that does not specify which limit was hit?

QWhy might a single self-hosted vLLM cluster on H100s outperform a hosted provider's rate-limited tier for long-context workloads?

QWhat happens to the TPM counter when prefix caching gives you a 0.1x cache read?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating TPM and RPM as redundant. They throttle completely different axes; your workload pattern decides which one bites first.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.