TPM and RPM each throttle a different axis, which one caps which?

Match pairs·Easy·4.0 · 0·~2 min·Asked atOpenAI Siemens Turing

Attempt it

Drag each answer to line up with its matching prompt

TPM

Workload that tends to hit the RPM limit first, even with low token volume.

RPM

Tokens Per Minute, the cap on total tokens (prompt + completion) submitted in a 60 second window.

Many small calls

Workload that tends to hit the TPM limit first, even at very low request counts.

Few long context calls

Requests Per Minute, the cap on the number of API calls, regardless of how big each one is.

Hosted LLM APIs throttle traffic on two independent axes: tokens and requests. **TPM** stands for **Tokens Per Minute**. It caps the total number of tokens (prompt plus completion) that a single account or organization can submit in any rolling 60-second window. A request with a 50k-token prompt counts as 50k tokens against this limit, even if it is a single API call. **RPM** stands for **Requests Per Minute**. It caps the number of API calls in the same 60-second window, regardless of how large each call is. One hundred tiny five-token requests counts the same as one massive 50k-token request. Which limit you hit first depends entirely on your workload shape. A chat product making thousands of short turns per minute (10-200 tokens each) hits the RPM ceiling well before TPM. A document-summarization batch job processing a few 100k-token PDFs per minute hits TPM long before RPM. Providers like OpenAI and Anthropic publish both numbers in their pricing tiers. As you spend more, both caps usually scale together, but they are tracked separately. Production systems that ignore one or the other end up surprised when traffic ramps and 429 errors arrive from an unexpected direction.

**TPM and RPM** are the two independent rate-limit axes that every hosted LLM API uses. **TPM = Tokens Per Minute.** Caps the total tokens (input plus output) per 60-second rolling window. Workloads that send long prompts (RAG, document analysis, code-base review) or generate long outputs (creative writing, long-form summarization) push this ceiling first. **RPM = Requests Per Minute.** Caps the count of API calls per 60-second window, independent of token volume. Workloads with many short calls (chat turns, classifier shots, tool-call dispatch, A/B testing) push this ceiling first. A 100-token chat turn and a 50,000-token document analysis both consume 1 RPM unit, but the document consumes 500x the TPM. Conversely, 500 short chat turns burn 500 RPM units but might only burn 50,000 TPM, while a single long-context call burns 50,000 TPM but only 1 RPM. Providers publish both: OpenAI's tier system (Tier 1-5) and Anthropic's organization-level rate-limit tiers each scale TPM and RPM together as account spend grows. Production engineering implications: - Monitor both metrics independently with separate alerts. - Use exponential backoff plus jitter on 429s, distinguishing TPM-limit retries (which need longer waits because tokens-consumed clears slowly) from RPM-limit retries. - For RPM-pressured chat workloads, batch where possible. For TPM-pressured long-context workloads, use prefix caching (Anthropic 0.1x reads, OpenAI 0.5x reads) which reduces actual token billing but does NOT change the TPM counter on most providers (verify this per provider, it changes).

TL;DR

TPM caps total tokens per 60s window; RPM caps API calls per 60s window. Many small calls hit RPM first; few long-context calls hit TPM first.

Memory aid

Easy to grasp

Imagine a bakery with two limits: one says 'no more than 200 customers per hour' and another says 'no more than 1000 loaves served per hour'. A coffee shop with hundreds of tiny one-loaf orders hits the customer cap first. A wedding planner ordering 50 huge multi-loaf platters at once hits the loaf cap first. Hosted LLM providers do the same thing: RPM is the customer count, TPM is the loaf count, and which one blocks you first depends entirely on your workload shape.

Key concepts

API rate limiting Tokens per minute Requests per minute Pricing and quotas Serving

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

TPM and RPM are the two acronyms every production LLM engineer needs to internalize before scaling an application beyond a few requests per second. They look superficially similar (both are per-minute caps), but they throttle structurally different axes and bite different workload shapes. Getting them confused in capacity planning leads to surprising 429s in production and burned engineering time chasing the wrong fix.

This deep dive walks through what each cap protects, how workload shape determines which one binds first, how to instrument and respond to 429s correctly, the prefix-caching interaction that is often misunderstood, and the self-hosted analogues in vLLM and SGLang.

What each cap actually counts

TPM counts tokens. Every prompt token sent and every completion token returned adds to a rolling 60-second counter. A request with a 50K-token prompt that produces a 5K-token completion adds 55K to the TPM counter for that minute. The counter is per-account or per-organization, not per-API-key.

RPM counts requests. Each API call adds 1 to a separate rolling 60-second counter, regardless of token volume. A 5-token request and a 50K-token request each add 1.

These counters are independent. Hitting one does not affect the other. A 429 response from a provider always indicates one specific cap was exceeded, and the better APIs include a header like x-ratelimit-limit-tokens versus x-ratelimit-limit-requests to disambiguate.

Tier-based scaling

Both OpenAI and Anthropic publish tier tables where TPM and RPM scale together as account spend climbs. OpenAI's Tier 1 might be 200 RPM and 40K TPM for GPT-5.5, while Tier 5 reaches millions of TPM and tens of thousands of RPM. The TPM-to-RPM ratio at higher tiers is typically larger than at lower tiers, reflecting the empirical reality that high-spend customers tend to lean long-context.

Workload shape and which cap binds first

Instrumentation, retries, and the prefix-cache subtlety

Self-hosted analogues and architecture decisions

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Property	TPM	RPM
What it caps	Total tokens per 60s window	API calls per 60s window
Counts toward limit	prompt_tokens + completion_tokens	Each API call equals 1 unit
Workload that hits first	Long-context, RAG, document analysis	Chat turns, tool-call dispatch, classifier
Self-hosted analogue (vLLM)	max_num_batched_tokens	max_num_seqs

Real products, models, and research that use this idea.

OpenAI's Tier 1-5 system scales both TPM and RPM with account spend; Tier 5 grants on the order of millions of TPM for GPT-5.5.
Anthropic publishes organization-level TPM and RPM for Claude Opus 4.7 and Sonnet 4.6, with separate caps for input tokens and output tokens.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you architect retries against a 429 response that does not specify which limit was hit?

Use the retry-after header when present. If absent, distinguish heuristically: a 429 right after a small request is probably RPM, after a large one probably TPM. Apply exponential backoff with jitter, with longer initial wait for suspected TPM (because the token counter clears over the full minute window) than for RPM.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating TPM and RPM as redundant. They throttle completely different axes; your workload pattern decides which one bites first.

60 second bullets to scan on the way to the call.

Expansion of TPM and RPM
Which axis each acronym throttles (tokens vs requests)

Primary sources. Browse if you want the original framing.