Rate limit strategies for high-throughput LLM applications

Scaling LLM calls past a few requests per second turns provider rate limits from a footnote into your primary constraint. Effective rate limit strategies high-throughput llm applications treat quotas as a scheduling problem: you must shape traffic, retry selectively, and degrade gracefully when an upstream throttles. This guide lays out an ordered path from measurement to resilient execution that you can implement this week.

1. Map the limit envelope before writing code

Every provider exposes limits differently. OpenAI publishes requests per minute (RPM) and tokens per minute (TPM) separately; Anthropic uses concurrent requests and daily token caps; self-hosted vLLM instances choke on KV-cache pressure long before any numeric quota. Pull the response headers on a real call and record them. Do not guess from dashboard screenshots.

import httpx

async def probe_limits():
    async with httpx.AsyncClient() as c:
        r = await c.post(
            "https://api.openai.com/v1/chat/completions",
            json={"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "hi"}]},
            headers={"Authorization": "Bearer $KEY"},
        )
        print("limit-req:", r.headers.get("x-ratelimit-limit-requests"))
        print("rem-tok:", r.headers.get("x-ratelimit-remaining-tokens"))

If you sit behind a gateway such as n4n.ai, the same OpenAI-compatible endpoint aggregates 240+ models but still returns per-provider limit signals after automatic fallback. Read those headers on the first call and cache them with a short TTL.

The tradeoff is that TPM limits bite harder than RPM for long outputs. A 10-request burst with 4k output tokens each can exhaust a 40k TPM ceiling instantly while leaving RPM untouched. Size max_tokens conservatively and measure your actual token shape with a production-like sample.

2. Bound concurrency with a semaphore

Unbounded async calls will trip 429s within milliseconds of a traffic spike. Use a semaphore sized to your measured concurrent limit, not your optimistic throughput target.

import asyncio

class LLMClient:
    def __init__(self, max_concurrent: int):
        self.sem = asyncio.Semaphore(max_concurrent)

    async def complete(self, payload: dict):
        async with self.sem:
            return await self._raw_call(payload)

Set max_concurrent to 70% of the documented concurrent cap. This leaves headroom for retries, health checks, and dashboard polling. A common pitfall is sizing the semaphore per process but running eight workers; multiply by worker count or you will silently oversubscribe.

For dynamic environments, adjust the semaphore from a sliding window of observed 429 rates. If 5% of calls in the last minute were throttled, drop concurrency by 10%. This closed-loop control beats static config.

3. Retry with exponential backoff and full jitter

A 429 is not a failure; it is a signal to wait. But naive fixed sleeps cause retry storms when many clients sync up after a provider restore.

import asyncio, random

class RateLimitError(Exception):
    def __init__(self, retry_after=None):
        self.retry_after = retry_after

async def call_with_backoff(fn, max_attempts=5):
    base = 0.2
    for attempt in range(max_attempts):
        try:
            return await fn()
        except RateLimitError as e:
            if attempt == max_attempts - 1:
                raise
            cap = e.retry_after or base * (2 ** attempt)
            sleep = random.uniform(0, cap)
            await asyncio.sleep(sleep)

Honor the Retry-After header if present; otherwise exponential with full jitter. Cap total wait at 30 seconds. Never retry 4xx other than 429/408; retrying 400 wastes quota and masks bugs.

Add a circuit breaker for persistent degradation. If a provider returns 429 on every call for ten seconds, stop sending and shed load to a fallback or queue.

4. Queue and prioritize instead of dropping

When the semaphore is full, push work into a priority queue rather than returning errors. High-priority user-facing requests jump ahead of batch embeddings.

import asyncio

queue: asyncio.PriorityQueue = asyncio.PriorityQueue()

async def worker(client):
    while True:
        prio, item = await queue.get()
        await client.complete(item)
        queue.task_done()

For multi-process deployments, use Redis or a broker with similar priority semantics. Tradeoff: queues add tail latency. Set a TTL on queued items; an expired prompt answered late is worse than a fast 429.

Batching is another lever. Many providers accept multiple messages in one request for embedding or classification; that converts N RPM hits into one. Do not batch interactive chat—users hate waiting.

5. Use fallback and routing directives

Single-provider dependencies break under regional outages. If your gateway supports automatic fallback when a provider is rate-limited or degraded, enable it, but still cap client concurrency because fallback consumes the same global TPM pool. You can also pin a model with a routing hint to keep cache locality.

{
  "model": "gpt-4o",
  "messages": [{"role": "user", "content": "Summarize this"}],
  "stream": false
}

With an OpenAI-compatible gateway that honors client routing directives and forwards provider cache-control hints, the same body hits prompt cache more often, reducing effective TPM. Do not assume fallback hides all limits; it shifts them to a secondary provider that may have tighter quotas.

6. Meter per-token usage and alert

Rate limits are proxies for cost and capacity. Track usage per route and per tenant:

from collections import defaultdict
usage = defaultdict(int)

def on_response(resp):
    usage[resp.model] += resp.usage.total_tokens

Wire this to metrics. Gateways with per-token usage metering simplify allocation, but you still need client-side counters to detect a single tenant burning the shared bucket. Set alerts at 80% of projected limit and shed low-priority traffic before hard throttling.

7. Common pitfalls that silently kill throughput

• Blocking the event loop with time.sleep in async code. Always use asyncio.sleep.
• Treating RPM and TPM as independent; a small request count with huge outputs still hits TPM.
• Retrying idempotency-unsafe writes. LLM calls are usually read-only, but tool calls that mutate state are not.
• Ignoring provider clock skew on Retry-After; clamp to your own max.
• Spawning unlimited queues without backpressure; memory grows until OOM.
• Forgetting that streaming responses hold connections; concurrent stream count is a limit too.

Pick two or three of these rate limit strategies high-throughput llm teams actually ship, implement them in order, and measure p99 latency under load before adding more complexity. Resilience comes from disciplined scheduling, not from hoping the provider raises your quota.