Token-per-minute vs requests-per-minute limits explained

Most LLM gateways enforce either token-per-minute vs requests-per-minute limits, and the difference dictates how you architect batch jobs, streaming, and fallback paths. A request cap counts the number of API calls; a token cap counts the sum of prompt, completion, and context tokens flowing through those calls. Pick the wrong mental model and you either throttle unnecessarily or blow your budget on a single massive prompt.

Capabilities: what each limit measures

The core of token-per-minute vs requests-per-minute limits is the resource they protect. RPM is a count of HTTP requests completed (or attempted) within a 60-second sliding window. TPM is the aggregate token count—usually prompt tokens plus completion tokens—attributed to your account in that same window.

Request counting

RPM ignores payload size. A 10-token query and a 100k-token document summarization both consume exactly one request against your RPM quota. This makes RPM a proxy for connection overhead and scheduler pressure on the provider side.

Token counting

TPM reflects computational load. A single request with 32k context eats 32k tokens from your TPM budget but only 1 from RPM. Most OpenAI-compatible APIs count both input and output tokens, though some older endpoints counted only completions. Always check the usage object.

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")

def approx_tokens(text: str) -> int:
    return len(enc.encode(text))

# Pre-flight estimate before sending
est = approx_tokens(system_prompt) + approx_tokens(user_msg)

Cost model and metering

RPM limits rarely map to cost; they exist to keep the gateway’s request router from falling over. TPM limits align with GPU memory and compute time, and therefore with billing. Per-token metering lets you attribute spend precisely to a tenant or feature.

When a gateway meters per-token usage, you can reconcile TPM limits with invoice line items. The response payload exposes the exact counts:

{
  "usage": {
    "prompt_tokens": 1200,
    "completion_tokens": 300,
    "total_tokens": 1500
  }
}

If you are evaluating token-per-minute vs requests-per-minute limits for cost control, TPM is the lever that matters. RPM only becomes a cost factor indirectly when you must shard work into more calls to stay under TPM, increasing request overhead.

Latency and throughput behavior

Under an RPM limit, you maximize throughput by parallelizing many small requests until you hit the request ceiling. Under TPM, a few large requests saturate the token budget quickly, leaving RPM headroom but zero token headroom.

A practical observation: streaming does not reduce TPM—tokens are counted as they are generated. It reduces time-to-first-token but not the total counted.

curl -i https://api.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"hi"}]}'
# HTTP/1.1 429 Too Many Requests
# X-RateLimit-Limit: 20000
# X-RateLimit-Remaining: 0
# X-RateLimit-Reset: 1700000000

The Reset epoch tells you when the window slides; backoff should target that, not a fixed sleep.

Ergonomics for client code

RPM is trivial to track: a counter and a timestamped window. TPM requires token estimation before send and adjustment after response. Underestimate and you get 429s mid-stream; overestimate and you underutilize quota.

A minimal TypeScript throttle that respects both:

class RateLimit {
  private reqs = 0;
  private tokens = 0;
  private windowStart = Date.now();
  constructor(private maxRpm: number, private maxTpm: number) {}

  async acquire(estTokens: number): Promise<void> {
    this.slide();
    if (this.reqs >= this.maxRpm || this.tokens + estTokens > this.maxTpm) {
      throw new Error("rate limited");
    }
    this.reqs++;
    this.tokens += estTokens;
  }

  release(usedTokens: number, estTokens: number) {
    this.tokens = Math.max(0, this.tokens - estTokens + usedTokens);
  }

  private slide() {
    if (Date.now() - this.windowStart >= 60_000) {
      this.reqs = 0;
      this.tokens = 0;
      this.windowStart = Date.now();
    }
  }
}

This is deliberately naive—production code should use a sliding log or token bucket—but it shows the extra bookkeeping TPM demands.

Ecosystem and gateway support

Foundational APIs (OpenAI, Anthropic, Mistral) publish both RPM and TPM, often per model tier. Gateways aggregate them across providers. An OpenRouter-class gateway like n4n.ai exposes both dimensions on one OpenAI-compatible endpoint covering 240+ models, and automatically falls back when a provider is rate-limited or degraded, but your client still must honor the stricter of the two quotas. It also forwards provider cache-control hints so repeated large contexts don’t double-count against TPM if the upstream supports caching.

For multi-provider routing, the effective limit is the minimum of your gateway quota and the underlying provider quota. Honor client routing directives: if you pin a model that has a lower TPM, your gateway’s higher TPM is irrelevant.

Hard limits and burst behavior

Providers differ on burst tolerance. RPM bursts are common—you might exceed 10% for a few seconds. TPM bursts are rarer because they map directly to VRAM allocation. A sudden 50k-token request can queue even if your average TPM is low.

Always read Retry-After and the rate-limit reset headers. Implement exponential backoff with jitter:

import random, time

def backoff(attempt: int, reset_epoch: int | None):
    if reset_epoch:
        sleep = max(0, reset_epoch - time.time())
    else:
        sleep = min(60, 2 ** attempt) + random.uniform(0, 1)
    time.sleep(sleep)

Side-by-side comparison

Which to choose: verdict by use case

Choosing between token-per-minute vs requests-per-minute limits depends on workload shape. They are not alternatives you toggle; they are simultaneous constraints. The verdict below tells you which one is binding for your scenario.

High-frequency small prompts

Classify millions of short strings? RPM is your binding constraint. Design for request concurrency, use HTTP keep-alive, and backoff on 429. Token limits are trivial because each call costs a few dozen tokens.

Long-document processing

Ingesting 100k-token PDFs? TPM dominates. Stream responses, cache context prefixes, and shard documents across requests to fit token windows. A single mega-request may exhaust your TPM for the whole minute.

Mixed workloads behind a gateway

When you route across multiple providers, treat the gateway’s published TPM and RPM as independent ceilings. Implement a token-aware queue that estimates before send. n4n.ai’s per-token metering and fallback helps absorb provider-side degradation, but you still need client-side throttling to avoid 429 storms.

Streaming chat at consumer scale

Each user turn is one request; RPM matters more. Token limits are secondary unless you allow huge system prompts. Budget per-user request rate, and use sticky sessions to avoid cross-region TPM fragmentation.

Batch inference jobs

TPM is the real limit. Pre-split tasks to fit token windows, use exponential backoff with jitter, and monitor usage.total_tokens to tune shard size. RPM will rarely bind if each shard is large.

In all cases, instrument both meters. Log remaining RPM and TPM from response headers, and alert when either drops below 20%. That visibility turns silent throttling into a tunable parameter.