Running a 429 error rate limit concurrency benchmark against an LLM inference endpoint reveals more about your client’s retry logic than the server’s true capacity. Most teams fire a fixed number of parallel requests, count the failures, and walk away with a misleading number. This analysis breaks down why sustained concurrent load testing demands modeling of token buckets, scoped limits, and fallback paths.
The myth of the simple concurrency test
A common mistake is to point wrk or a quick Python script at an endpoint with -c 100 and report the percentage of 429 responses. That number is meaningless without duration context. Rate limits are typically expressed as a rate (requests per minute, tokens per minute), not a connection cap. If you burst 100 requests in the first second against a 60 RPM limit, you will see 40+ rejections immediately, but the remaining 60 may sail through over the next 59 seconds.
Sustained load changes the picture. A proper 429 error rate limit concurrency benchmark holds a target request rate for minutes, not seconds, and observes how the system behaves after the initial bucket drains. Without that sustained phase, you cannot distinguish a provider that is truly saturated from one that simply rejects bursts.
What a 429 actually signals
Token buckets vs fixed windows
Most LLM providers implement a token bucket or leaky bucket variant. A bucket holds N request tokens, refills at R per second, and allows bursts up to N. A 429 means the bucket is empty right now, not that the model GPU is overloaded. Fixed-window limiters (e.g., “100 requests per rolling minute”) create boundary effects where a client can sneak through a burst at the window edge. Knowing which model your target uses dictates how you schedule load.
If you assume a fixed window and sleep until the next minute, you will mistime requests against a token bucket and artificially inflate your measured 429 error rate limit concurrency benchmark. Read the provider docs or infer from response headers (Retry-After, X-RateLimit-Remaining) before writing the test harness.
Per-key, per-model, per-route scoping
Limits are rarely global. A single API key may have 60 RPM for gpt-4o, 200 RPM for gpt-4o-mini, and separate caps for embeddings. Chat completions and completions endpoints often count separately. Concurrent load that mixes models produces a blended 429 rate that hides which specific limit you hit.
Isolate the variable. Run your benchmark against one model, one route, one key. Then repeat with mixtures only after you have baseline single-stream numbers.
Designing a meaningful 429 error rate limit concurrency benchmark
Model the client, not just the server
The server’s limit is fixed; your client’s behavior is what determines observed failure rate. A naive client that retries immediately on 429 amplifies load and triggers longer cooldowns. A client with exponential backoff and jitter converges to the admissible rate. Your benchmark must simulate the client you will actually ship.
Code: a minimal async load generator with backoff
The script below spawns a controlled number of concurrent workers, each looping until the test duration expires. It counts successes, 429s, and exhausted retries. It uses a semaphore to cap true concurrency and applies exponential backoff with jitter on 429.
import asyncio, aiohttp, random, time
BASE = "https://api.example.com/v1/chat/completions"
HEADERS = {"Authorization": "Bearer $KEY", "Content-Type": "application/json"}
PAYLOAD = {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "hi"}]}
async def call(session, sem, stats):
async with sem:
for attempt in range(5):
try:
async with session.post(BASE, json=PAYLOAD, headers=HEADERS) as resp:
if resp.status == 429:
stats['rate_limited'] += 1
await asyncio.sleep((2 ** attempt) + random.random())
continue
elif resp.status == 200:
stats['ok'] += 1
return
else:
stats['other'] += 1
return
except Exception:
stats['error'] += 1
return
stats['exhausted'] += 1
async def main(concurrency, duration):
sem = asyncio.Semaphore(concurrency)
stats = {'ok':0,'rate_limited':0,'other':0,'error':0,'exhausted':0}
async with aiohttp.ClientSession() as session:
end = time.time() + duration
tasks = []
while time.time() < end:
tasks.append(asyncio.create_task(call(session, sem, stats)))
await asyncio.sleep(0.01)
await asyncio.gather(*tasks)
print(stats)
asyncio.run(main(50, 300))Run this for five minutes at concurrency 50. The rate_limited counter tells you how many times the bucket rejected you; the ok counter tells you admitted throughput. The ratio after the first 30 seconds is your stabilized 429 error rate limit concurrency benchmark.
Measuring the right metric: admitted vs rejected
Raw 429 count is vanity. The metric that matters is admitted request rate—successful completions per minute after backoff stabilizes. Plot it over time. A healthy client approaches the provider’s published limit asymptotically. If admitted rate flattens below the limit while 429s keep climbing, your backoff is too aggressive or your concurrency too low to fill the bucket.
Tradeoffs of aggressive concurrency
Throughput vs error rate
Cranking concurrency to 500 does not multiply throughput past the limit. It multiplies 429s. Each rejected request costs a round trip and a context switch. In our local token-bucket simulations, raising concurrency from 10 to 200 against a 60 RPM bucket increased 429 share from near-zero to dominant while admitted rate stayed pinned at the bucket refill rate. The only real effect was higher client CPU and worse tail latency for the requests that did succeed because they queued behind retries.
The tradeoff is clear: concurrency should be tuned to slightly exceed the bucket depth, not the desired rate. That keeps the bucket fed without drowning in rejections.
Fallback and gateway behavior
If you route through an OpenAI-compatible gateway such as n4n.ai, which provides automatic fallback when a provider is rate-limited, the client-side 429 error rate limit concurrency benchmark will reflect failover success rather than raw provider limits. The gateway honors routing directives and forwards cache-control hints, so your test must account for that layer: a request that would 429 on provider A may succeed on provider B transparently. Benchmarking the gateway without modeling fallback understates the resilience you actually get.
Honest limitations of the method
No black-box benchmark can reveal the provider’s internal queueing. You are measuring the edge, not the GPU. A 429 may also be returned during provider degradation unrelated to your personal limit; distinguishing the two requires correlating with Retry-After and status of the provider’s status page. Additionally, token-based limits (tokens per minute) matter more for long outputs than request counts; a benchmark using tiny prompts undercounts token pressure.
Decisive takeaway
Treat a 429 error rate limit concurrency benchmark as a test of your client’s capacity to gracefully absorb rate shaping, not a probe of server max throughput. Use sustained load, isolate model and route, implement backoff with jitter, and measure admitted rate after stabilization. If you sit behind a gateway with fallback, include that failover logic in the harness or you will misread the results. Ship the client that fills the bucket quietly, not the one that floods and retries.