Scaling concurrent users: where API rate limits break down

Most teams treat provider quotas as a static ceiling, but scaling concurrent users API rate limits reveals a messier truth: the limits are measured at the account level while consumption happens across hundreds of independent request paths. Under real concurrency, you exhaust limits from burst shape, not total volume.

The false assumption behind provider rate limits

Providers expose rate limits as requests per minute (RPM) and tokens per minute (TPM) tied to an API key. The mental model is a single client pacing itself. That holds for a cron job or a prototype. It collapses the moment you put real users behind a shared key.

The limit is a global counter. If you have 500 concurrent users, each triggering one chat completion, the requests land within the same sliding window. The provider does not know about your users; it sees a flood from one credential.

Worse, token limits are even more deceptive. A single long completion can consume 30% of your TPM budget. While that request is in flight, every other concurrent user is competing for the remaining sliver. Your application code thinks in terms of “one user, one request”; the provider thinks in terms of “one key, one leaky bucket.”

What actually happens at 1,000 concurrent users

Consider a support chat app. Each user action fires three parallel calls: a classifier, a retrieval rerank, and the main answer. At 300 active users, that is 900 requests per action cycle. If cycles average every 8 seconds, you are attempting ~110 requests per second.

A typical mid-tier account might allow a few hundred RPM. The math says you are fine on average. The reality is a sawtooth: at the top of the second, 900 requests arrive simultaneously. The provider’s edge rejects the overflow with HTTP 429.

Suppose your limit is 100k TPM. A 1k-token prompt with a 500-token response uses 1.5k tokens. If 300 users hit at once, you just requested 450k tokens in one second—4.5x your per-minute budget. The limit is per minute, but the burst is per second.

import asyncio, openai

client = openai.AsyncOpenAI(api_key="sk-...")

async def handle_user(user_id: str, prompt: str):
    # three fan-out calls
    await asyncio.gather(
        client.chat.completions.create(model="gpt-4o-mini", messages=[{"role":"user","content":prompt}]),
        client.chat.completions.create(model="gpt-4o-mini", messages=[{"role":"user","content":"classify: "+prompt}]),
        client.chat.completions.create(model="gpt-4o-mini", messages=[{"role":"user","content":"rerank: "+prompt}]),
    )

# 300 users hit at once
async def main():
    await asyncio.gather(*[handle_user(str(i), "hello") for i in range(300)])

Run this against a shared key with a modest limit and you will watch RateLimitError propagate within milliseconds. The failures are correlated: all three calls for a user fail together, and retries amplify the spike.

Bursts, fan-out, and the thundering herd

The naive fix is to catch 429 and retry immediately. That turns a spike into a sustained storm. Every client retries at the same time because they all got rejected at the same time.

Add exponential backoff without jitter and you get a synchronized second wave. The provider’s limit resets, all retries land in the first 100ms of the window, and you are back to square one.

async def call_with_retry(fn, attempts=5):
    for i in range(attempts):
        try:
            return await fn()
        except openai.RateLimitError:
            await asyncio.sleep(2 ** i)  # no jitter: synchronized retry

This is where scaling concurrent users API rate limits stops being a config issue and becomes a distributed systems problem. You are building a client-side scheduler for a resource you do not control.

Why per-user limits don’t map to provider limits

Product specs often say “each user can send one message per second.” If your provider grants 200 RPM, that is 3.3 requests per second globally. You can support three users at that SLA, not 200. The mismatch is fundamental: provider limits are aggregate, user expectations are per-entity.

You must either negotiate a higher quota, multiplex users onto a shared budget with admission control, or push work to background queues. None of these are visible in the provider dashboard.

Three patterns that survive contact with reality

Local token buckets per process

Treat the provider limit as a shared resource that your own process must ration. A token bucket implemented in-process lets you cap outgoing request rate and token estimate before sending.

import asyncio, time

class TokenBucket:
    def __init__(self, rate: float, capacity: float):
        self.rate = rate  # tokens per second
        self.capacity = capacity
        self.tokens = capacity
        self.last = time.monotonic()
        self.lock = asyncio.Lock()

    async def consume(self, needed: float):
        async with self.lock:
            now = time.monotonic()
            self.tokens += (now - self.last) * self.rate
            self.tokens = min(self.tokens, self.capacity)
            self.last = now
            if self.tokens < needed:
                wait = (needed - self.tokens) / self.rate
                await asyncio.sleep(wait)
                self.tokens = 0
            else:
                self.tokens -= needed

You still need to size the bucket below the provider’s real limit, leaving headroom for other processes and for token variance.

Semaphore-based concurrency caps

Even with rate limiting, unbounded concurrency will exhaust file descriptors and memory. A simple semaphore bounds in-flight calls.

sem = asyncio.Semaphore(20)  # max 20 concurrent LLM calls per process

async def bounded_call(fn):
    async with sem:
        return await call_with_jitter_retry(fn)

Combine the semaphore with a jittered backoff:

import random

async def call_with_jitter_retry(fn, attempts=5):
    for i in range(attempts):
        try:
            return await fn()
        except openai.RateLimitError:
            await asyncio.sleep((2 ** i) * random.uniform(0.5, 1.5))

Degrade, don’t block

When the limit is saturated, serve a cached answer, a smaller model, or a static fallback. The user experiences latency or reduced quality, not a hard error.

async def handle_user_safe(user_id, prompt):
    try:
        return await bounded_call(lambda: client.chat.completions.create(model="gpt-4o", messages=...))
    except openai.RateLimitError:
        return await client.chat.completions.create(model="gpt-4o-mini", messages=...)  # cheaper tier

The gateway escape hatch

An inference gateway can absorb some of this complexity. A gateway such as n4n.ai exposes a single OpenAI-compatible endpoint across 240+ models and will automatically route around a degraded provider, but your client still needs to handle the 429 from the gateway when its own upstream is saturated. The gateway does not invent capacity; it trades one provider’s limit for another’s, and may switch models under you. Your application code must declare which model substitutions are acceptable via routing directives.

If you honor provider cache-control hints at the gateway layer, repeated prefixes cost fewer tokens, which indirectly relieves TPM pressure. That is a real win, but it does not change the concurrency math.

Tradeoffs you can’t avoid

Client-side throttling reduces the chance of 429s but adds latency. A token bucket sized for the worst-case window means idle capacity most of the time. Semaphores protect your process but cap throughput per instance; you scale horizontally, which multiplies the number of buckets all hitting the same global limit.

Multi-provider fallback improves resilience at the cost of observability. When a request silently moves from a frontier model to a smaller one, your eval metrics shift. You need explicit logging of which model actually served the response.

Queueing requests decouples ingestion from processing, but users hate waiting. A 30-second queue for a chat message is unacceptable; a 2-second queue for a background summarization job is fine. The right answer depends on the UX contract, not the API.

Per-token metering (as offered by some gateways) gives you precise cost attribution per user, which helps you find the heavy tenants that are starving the rest. But metering is after the fact—it does not prevent the 429.

Takeaway

Scaling concurrent users API rate limits is not about raising the quota; it is about shaping traffic to match a global counter that was never designed for your fan-out. Implement local token buckets, bound concurrency with semaphores, jitter every retry, and treat model fallback as a planned degradation path rather than a surprise.

If you deploy behind a gateway, use its routing hints and cache controls, but keep your own backpressure. The teams that survive a 10x user spike are the ones who designed for the burst, not the average.