Stress testing LLM endpoints for rate limit behavior

Stress testing LLM rate limits is the only way to know how your application behaves when a provider starts rejecting requests. This guide walks through building a reproducible harness that pushes an OpenAI-compatible endpoint into 429 territory and validates your client’s resilience. You will learn to measure limit thresholds, implement backoff, and confirm fallback paths without guessing.

Step 1: Establish your rate limit baseline

Before sending traffic, capture the limits your endpoint enforces. Most providers return headers like x-ratelimit-limit-requests and x-ratelimit-remaining-requests on each response. If you use a gateway, those headers may reflect the aggregate or per-provider view.

Run a single curated call and inspect headers:

curl -i https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'

Look for x-ratelimit-limit-requests: 500 or similar. If the endpoint is a gateway that aggregates 240+ models, the limit might be per model or per key. Document the observed numbers; you will ramp past them in later steps.

If the provider does not send limit headers, infer the threshold with a binary search: send 10 requests/sec, then 20, then 40, until you see the first 429. Record that knee. Token-based limits are trickier—watch x-ratelimit-limit-tokens and the usage block in responses.

Step 2: Build a concurrent request generator

A real client hits the endpoint from many threads. Use Python with asyncio and httpx to fire parallel chat completion requests. Keep the payload tiny to avoid token limits masking request limits.

import asyncio, httpx, os

ENDPOINT = "https://api.openai.com/v1/chat/completions"
HEADERS = {
    "Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}",
    "Content-Type": "application/json",
}
PAYLOAD = {
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "ping"}],
    "max_tokens": 5,
}

async def call_once(client, sem):
    async with sem:
        r = await client.post(ENDPOINT, headers=HEADERS, json=PAYLOAD)
        return r.status_code

async def run_batch(concurrency, total):
    sem = asyncio.Semaphore(concurrency)
    async with httpx.AsyncClient(timeout=30.0) as client:
        tasks = [call_once(client, sem) for _ in range(total)]
        return await asyncio.gather(*tasks)

This skeleton returns status codes. In stress testing LLM rate limits, you want to count 200s versus 429s directly. Set a client timeout so a stalled connection does not hang the test. Reuse a single AsyncClient to benefit from connection pooling; otherwise you will measure socket exhaustion instead of API limits.

Step 3: Ramp concurrency until you trigger 429s

Start below the documented limit, then increase. A simple loop from 10 to 500 concurrent workers exposes the knee.

import collections

async def ramp_test():
    results = []
    for conc in [10, 50, 100, 200, 500]:
        codes = await run_batch(conc, conc)
        counts = collections.Counter(codes)
        results.append((conc, counts))
        print(conc, dict(counts))

When counts[429] > 0, you have exceeded the request rate. Note the concurrency level where the first 429 appears. That is your empirical request limit, which may differ from the header if the provider uses token-based throttling.

During stress testing LLM rate limits, also watch latency. A provider may queue requests instead of rejecting them, causing p95 latency to spike before 429s appear. Log r.elapsed for each call. If 200s come back but take 10x baseline, you are hitting a soft limit. Adjust the ramp to include a token-heavy payload (max_tokens: 2000) to test token throughput separately.

Step 4: Implement exponential backoff with jitter

A client that crashes on 429 is unacceptable. Wrap the call in a retry loop that respects Retry-After if present, otherwise uses exponential backoff.

import random

async def call_with_backoff(client, sem, max_retries=5):
    async with sem:
        for attempt in range(max_retries):
            r = await client.post(ENDPOINT, headers=HEADERS, json=PAYLOAD)
            if r.status_code == 200:
                return r.json().get("usage", {})
            if r.status_code == 429:
                retry_after = r.headers.get("retry-after")
                wait = float(retry_after) if retry_after else (2 ** attempt) + random.random()
                await asyncio.sleep(wait)
                continue
            r.raise_for_status()
    return None

Run the same ramp with this function. Success means the batch eventually completes with mostly 200s and no unhandled exceptions. The jitter (random.random()) prevents a thundering herd when many workers retry simultaneously. Without jitter, synchronized retries deepen the penalty.

Step 5: Verify fallback when a provider is degraded

If you route through a gateway such as n4n.ai, which offers an OpenAI-compatible endpoint across 240+ models with automatic fallback when a provider is rate-limited, your test must exercise that path. The gateway honors client routing directives and forwards provider cache-control hints, so you can force a primary provider into a throttled state and observe recovery.

Using a client that sends routing headers:

HEADERS_WITH_ROUTE = {
    **HEADERS,
    "x-n4n-route": "primary=openai; fallback=anthropic",  # example directive format
    "cache-control": "max-age=3600",  # forwarded to provider for cache hits
}

Fire a batch that exceeds OpenAI’s request limit. If the gateway works, you will see 200 responses with usage objects even though the primary would have 429’d. Without a gateway, simulate fallback by catching 429 and switching ENDPOINT to a secondary key or provider in your own code. The point is to prove the request completes, not to test the gateway internals.

Step 6: Validate per-token metering

Rate limits are often token-based, not just request-based. After a successful call, inspect the usage field:

{
  "usage": {
    "prompt_tokens": 3,
    "completion_tokens": 5,
    "total_tokens": 8
  }
}

If your gateway performs per-token usage metering, sum total_tokens across the batch and compare against the limit header x-ratelimit-limit-tokens. Stress testing LLM rate limits should push token throughput, not only request count. Use a larger max_tokens in the payload to burn tokens faster.

A quick metering check:

async def meter_batch(n):
    sem = asyncio.Semaphore(20)
    async with httpx.AsyncClient() as client:
        usages = [await call_with_backoff(client, sem) for _ in range(n)]
    total = sum(u.get("total_tokens", 0) for u in usages if u)
    return total

Print total and confirm it is close to n * avg_tokens. If the gateway reports usage per request, reconcile the sums in your test output.

Step 7: Automate and define success criteria

Wrap the ramp and backoff in a pytest or CI job. Set explicit pass/fail conditions:

• The harness triggers at least one 429 at concurrency X.
• After backoff, ≥95% of requests ultimately return 200.
• Fallback (if configured) yields 200 when primary is saturated.
• Total metered tokens match expected sum within 1% margin.

A minimal verification script:

async def test_rate_limit_behavior():
    codes = await ramp_test()
    assert any(429 in c.values() for _, c in codes)  # limits hit
    total = await meter_batch(50)
    assert total > 0
    sem = asyncio.Semaphore(50)
    async with httpx.AsyncClient() as client:
        usages = [await call_with_backoff(client, sem) for _ in range(50)]
    assert all(u is not None for u in usages)

Run it in CI on a schedule. Providers change limits without notice; continuous stress testing LLM rate limits catches regressions before users do. Store the baseline numbers in a versioned file and fail the build if the empirical limit drops by more than 20%.

How to verify success

You have a working test when the following hold:

1. You can reproduce 429s on demand by setting concurrency above the baseline.
2. Your backoff logic recovers the batch without manual intervention.
3. If a gateway is in the path, fallback engages and returns valid completions.
4. Token metering matches the provider’s reported usage.

Treat the harness as production code. Version it, parameterize the endpoint, and alert on unexpected 200→429 ratio shifts. That discipline turns rate limit surprises into scheduled, boring test runs.