Tokens per minute vs requests per minute limits explained

When you hit a rate limit on an LLM API, the error message rarely tells the full story. The interaction between tokens per minute vs requests per minute determines whether your batch job stalls or your chat endpoint returns 429s, and the binding constraint shifts dramatically with prompt length, output length, and concurrency. Most providers enforce both simultaneously, and the one that bites you first depends entirely on your traffic shape.

What each limit actually measures

Requests per minute (RPM) counts API calls. Every POST /v1/chat/completions increments the counter, whether you send 10 tokens or 10,000. Tokens per minute (TPM) counts the sum of prompt tokens plus completion tokens across all requests in a rolling minute window. A single 100k-token request consumes the same TPM budget as one thousand 100-token requests, but only 1 RPM versus 1,000 RPM.

Providers expose both because they protect different resources. RPM protects control-plane throughput — authentication, routing, request validation, and per-request overhead in the inference stack. TPM protects GPU memory bandwidth and compute allocation. You hit RPM limits when you send many small requests. You hit TPM limits when you send fewer large requests or long completions.

// Typical rate limit headers from a provider
{
  "x-ratelimit-limit-requests": "3000",
  "x-ratelimit-limit-tokens": "1000000",
  "x-ratelimit-remaining-requests": "2847",
  "x-ratelimit-remaining-tokens": "987654",
  "x-ratelimit-reset-requests": "45.2",
  "x-ratelimit-reset-tokens": "52.1"
}

The binding constraint formula

For any given workload, you can calculate which limit binds first. Let avg_prompt_tokens be your average input length, avg_completion_tokens your average output length, and concurrency your sustained requests per minute. Then:

rpm_consumed = concurrency
tpm_consumed = concurrency * (avg_prompt_tokens + avg_completion_tokens)

The binding limit is whichever ratio hits 1.0 first:

rpm_ratio = rpm_consumed / rpm_limit
tpm_ratio = tpm_consumed / tpm_limit

If rpm_ratio > tpm_ratio, RPM binds. If tpm_ratio > rpm_ratio, TPM binds. The crossover point occurs when:

avg_prompt_tokens + avg_completion_tokens = tpm_limit / rpm_limit

For a typical tier offering 3,000 RPM and 1,000,000 TPM, the crossover is ~333 tokens per request. Below that, RPM binds. Above it, TPM binds. This means a chat application with 200-token exchanges is RPM-constrained, while a document summarization pipeline with 8,000-token inputs is TPM-constrained — even at the same request rate.

How burst behavior differs

RPM and TPM windows reset independently, often on different schedules. RPM typically uses a fixed or sliding window per minute. TPM frequently uses a token bucket with per-second refill, allowing short bursts above the nominal per-minute rate if you’ve been idle.

# Token bucket simulation for TPM burst behavior
class TokenBucket:
    def __init__(self, capacity: int, refill_per_second: float):
        self.capacity = capacity
        self.tokens = capacity
        self.refill_per_second = refill_per_second
        self.last_refill = time.monotonic()

    def take(self, n: int) -> bool:
        self._refill()
        if self.tokens >= n:
            self.tokens -= n
            return True
        return False

    def _refill(self):
        now = time.monotonic()
        elapsed = now - self.last_refill
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_per_second)
        self.last_refill = now

This means you can sometimes absorb a spike of large requests if you’ve been quiet, but a sustained stream of small requests will hit the RPM ceiling with no burst allowance. Design your client to respect both headers independently — don’t assume one implies headroom on the other.

Streaming and the token accounting trap

Streaming responses (stream: true) complicate TPM accounting. Most providers count tokens as they’re generated, not at request completion. If you open 50 concurrent streams each generating 2,000 tokens, you’ll consume 100,000 TPM over the minute — but the RPM cost is only 50. The tokens trickle out over seconds, so your TPM usage ramps gradually rather than spiking at request start.

However, some providers pre-charge estimated max tokens at request initiation for streaming calls, then refund the difference on completion. Check your provider’s documentation. If they pre-charge, streaming offers no TPM advantage over non-streaming. If they charge incrementally, streaming smooths TPM consumption and can help you stay under the limit during bursts.

# Client-side TPM pacing for streaming workloads
async def paced_stream(client, messages, max_tokens, tpm_budget_per_second):
    """Yield tokens while respecting a per-second TPM budget."""
    tokens_used = 0
    start = time.monotonic()
    
    async for chunk in client.chat.completions.create(
        messages=messages, max_tokens=max_tokens, stream=True
    ):
        chunk_tokens = estimate_tokens(chunk.choices[0].delta.content or "")
        tokens_used += chunk_tokens
        
        # Pace if we're exceeding budget
        elapsed = time.monotonic() - start
        allowed = elapsed * tpm_budget_per_second
        if tokens_used > allowed:
            await asyncio.sleep((tokens_used - allowed) / tpm_budget_per_second)
        
        yield chunk

Model-tier differences

Rate limits vary by model, not just by account tier. A provider might offer 10,000 RPM / 2,000,000 TPM on a small model but only 500 RPM / 200,000 TPM on their largest model. The crossover point shifts accordingly. On the large model with a 400-token crossover, even moderate chat workloads become TPM-bound.

Multi-model routing — sending different request types to different models — requires tracking separate limit buckets per model. A router that balances load across models must maintain per-model RPM and TPM counters and route around exhausted buckets. This is where a gateway that exposes unified limit headers across providers simplifies client logic significantly.

Comparison table

Client-side strategies for each limit

When RPM binds, reduce request count. Batch multiple logical operations into single requests using few-shot prompts or structured output schemas that return multiple results. Use connection pooling and keep-alive to amortize TLS handshakes. If your provider supports it, use the n parameter to generate multiple completions per request — this costs 1 RPM but n times the TPM.

// Single request, multiple completions — saves RPM
{
  "model": "gpt-4o-mini",
  "messages": [{"role": "user", "content": "Generate 3 variations..."}],
  "n": 3,
  "max_tokens": 500
}

When TPM binds, reduce token volume. Compress prompts with retrieval-augmented generation instead of stuffing full documents. Set max_tokens conservatively — every token reserved counts against TPM even if unused. Enable provider-side caching where available (Anthropic’s prompt caching, OpenAI’s predicted outputs) to reduce billed tokens on repeated prefixes. Truncate conversation history aggressively for chat workloads.

# Prompt compression example: RAG vs full context
def build_prompt_rag(query: str, retriever, max_chunks: int = 3) -> str:
    """Retrieve only relevant chunks instead of full document."""
    chunks = retriever.search(query, k=max_chunks)
    context = "\n\n".join(c.text for c in chunks)
    return f"Context:\n{context}\n\nQuestion: {query}\nAnswer:"

def build_prompt_full(doc: str, query: str) -> str:
    """Naive full-document context — burns TPM."""
    return f"Document:\n{doc}\n\nQuestion: {query}\nAnswer:"

Monitoring and alerting

Track both ratios in your dashboards. Alert on max(rpm_ratio, tpm_ratio) > 0.8 for sustained periods. Break down by model, endpoint, and customer if you’re multi-tenant. The ratio that approaches 1.0 tells you which optimization lever to pull.

# PromQL: max utilization across both dimensions per model
max by (model) (
  max(
    rate(http_requests_total{status="429"}[5m]) / rate(http_requests_total[5m]),
    rate(tokens_consumed_total[5m]) / 60 / tpm_limit_per_model
  )
) > 0.8

Log the limit headers on every response. Correlate 429s with the specific header that hit zero. This distinguishes “too many chat messages” (RPM) from “context too long” (TPM) in your incident response.

Which to choose — verdict by use case

High-volume chat / customer support bots: RPM binds first. Optimize by batching unrelated user messages into single requests where latency permits, using n for candidate generation, and keeping conversation history short. Provision for peak RPM, not peak TPM.

Document processing / summarization / RAG with long contexts: TPM binds first. Optimize with aggressive retrieval (top-k instead of full docs), prompt compression, and output length limits. Use streaming with incremental token charging to smooth consumption. Provision for peak TPM.

Code generation / agent workflows with tool calls: Both bind depending on phase. Planning phases with long prompts hit TPM. Rapid tool-call loops hit RPM. Implement per-phase pacing: token-budgeted for planning, request-budgeted for execution.

Multi-model routing layer: Track both limits per model independently. Route around exhausted buckets. Expose unified retry-after to clients based on the tighter of the two limits for the selected model. This is the exact problem a gateway like n4n.ai solves by normalizing limit headers across 240+ models and automatically falling back when a provider’s bucket is exhausted.

Batch / async workloads: Neither binds if you pace correctly. Implement a token-bucket client that respects both RPM and TPM limits with configurable safety margins (80% of advertised limits). Process jobs in priority order, shedding low-priority work when limits approach.

The practical rule: measure your average tokens per request in production. If it’s below your provider’s TPM_limit / RPM_limit crossover, you’re RPM-constrained — optimize request count. If it’s above, you’re TPM-constrained — optimize token volume. Most teams discover they’re constrained by the limit they weren’t watching.