A 429 from an LLM provider means you’ve blown past a quota, but retrying harder just digs the hole deeper. Implementing client-side rate limiting avoid 429 errors at the application layer lets you shape traffic before it leaves your process, cutting wasted requests and tail latency. This guide gives you a token bucket, backoff logic, and a verification loop you can ship in Python or TypeScript today.
Step 1: Read the limit headers, not just the docs
Providers publish RPM (requests per minute) and TPM (tokens per minute), but the enforced values shift with tier and load. The authoritative signal is in response headers: x-ratelimit-limit-requests, x-ratelimit-remaining-requests, and retry-after on a 429. Parse them on every call so your limiter tracks reality, not a guess.
import httpx
def extract_limits(headers: dict) -> tuple[int, int] | None:
limit = headers.get("x-ratelimit-limit-requests")
remaining = headers.get("x-ratelimit-remaining-requests")
if limit and remaining:
return int(limit), int(remaining)
return NoneIf you call an OpenAI-compatible gateway, the same headers propagate. Use them to seed your bucket capacity instead of hardcoding from a stale wiki page.
Step 2: Implement a token bucket
The token bucket is the right primitive: it allows bursts up to capacity and refills at a steady rate. Below is a minimal async Python version using time.monotonic and asyncio.
import asyncio
import time
class TokenBucket:
def __init__(self, rate: float, capacity: float):
self.rate = rate # tokens per second
self.capacity = capacity
self.tokens = capacity
self.last = time.monotonic()
self.lock = asyncio.Lock()
async def acquire(self, needed: float = 1.0):
async with self.lock:
while True:
now = time.monotonic()
self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
self.last = now
if self.tokens >= needed:
self.tokens -= needed
return
wait = (needed - self.tokens) / self.rate
await asyncio.sleep(wait)TypeScript mirrors this with Date.now() and a promise-based sleep:
export class TokenBucket {
private tokens: number;
private last = Date.now();
constructor(private rate: number, private capacity: number) {
this.tokens = capacity;
}
async acquire(needed = 1): Promise<void> {
while (true) {
const now = Date.now();
this.tokens = Math.min(this.capacity, this.tokens + ((now - this.last) / 1000) * this.rate);
this.last = now;
if (this.tokens >= needed) {
this.tokens -= needed;
return;
}
const waitMs = ((needed - this.tokens) / this.rate) * 1000;
await new Promise(r => setTimeout(r, waitMs));
}
}
}Set rate = rpm / 60 and capacity = burst (e.g., 10% of RPM). Client-side rate limiting avoid 429 by never issuing more than the bucket allows, even when your upstream queue spikes.
Step 3: Wrap your LLM call
Inject the bucket into the request path. In Python, wrap httpx.AsyncClient or an OpenAI SDK client. Here we use raw httpx against an OpenAI-compatible endpoint:
async def chat(bucket: TokenBucket, payload: dict, client: httpx.AsyncClient):
await bucket.acquire(1) # 1 request token; add TPM tracking separately
resp = await client.post("/v1/chat/completions", json=payload)
if resp.status_code == 429:
# handled in step 4
pass
return respFor TypeScript with fetch:
async function chat(bucket: TokenBucket, payload: any): Promise<Response> {
await bucket.acquire(1);
return fetch("/v1/chat/completions", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(payload),
});
}If you route through a gateway such as n4n.ai, automatic fallback covers provider-side degradation, but your loop still needs client-side rate limiting avoid 429 from your own over-issuance.
Step 4: Back off on 429 with jitter
A 429 means the bucket estimate was wrong or a shared key got throttled. Read retry-after (seconds) and add full jitter to avoid thundering herd.
import random
import asyncio
async def call_with_backoff(bucket, payload, client, max_retries=5):
for attempt in range(max_retries):
resp = await chat(bucket, payload, client)
if resp.status_code != 429:
return resp
retry_after = float(resp.headers.get("retry-after", 1))
sleep = retry_after * (0.5 + random.random()) # jitter
bucket.rate *= 0.5 # shrink rate temporarily
await asyncio.sleep(sleep)
raise RuntimeError("exhausted retries on 429")The rate halving is critical: blind retry ignores the cause. Client-side rate limiting avoid 429 by adapting the refill rate downward when rejected, then recovering slowly after a cool-down.
Step 5: Track token usage, not just request count
Request limits are easy; token-per-minute limits bite harder on long prompts. Extend the bucket to deduct based on usage.total_tokens if your provider returns it. Most OpenAI-compatible responses include a usage JSON object.
if resp.status_code == 200:
used = resp.json().get("usage", {}).get("total_tokens", 0)
# log used; for TPM use a separate bucket acquired before sendImplement a second bucket for TPM with rate = tpm / 60 and acquire used / 1000 before sending. This dual-bucket pattern is what keeps you under both ceilings when prompt sizes vary.
Step 6: Verify with a controlled load test
You can’t claim success without evidence. Write a script that fires at 2x your assumed limit and assert zero 429s in the log.
python -m locust -f load_test.py --headless -u 200 -r 20 -t 2mIn load_test.py, reuse your production client wrapped with the bucket. After the run, check:
- Application logs show no
429responses. - p99 latency stays under your SLO (e.g., 2s).
- Bucket
ratenever exceeded the provider’s published RPM when averaged over a minute.
If you see 429s, your header parsing in Step 1 missed a quota type. Fix the seed, not the sleep.
Step 7: Emit metrics and alert
A limiter silent in dashboards is a limiter that drifts. Export tokens_waiting, rate_current, and 429_count to Prometheus. Alert if 429_count > 0 for five minutes—that means your client-side rate limiting avoid 429 strategy needs retuning because the provider changed limits.
from prometheus_client import Gauge, Counter
TOKENS_WAITING = Gauge("bucket_waiting", "tokens blocked")
RATE = Gauge("bucket_rate", "current token rate")
FOUR29 = Counter("http_429_total", "count of 429s")Wire these inside acquire and the backoff path so you can see throttling before users do.
Step 8: Handle cache-control to skip redundant calls
Some gateways forward provider cache-control hints. If your request hits a cached completion, you avoid both a token charge and a rate-limit deduction. Honor cache-control: max-age by skipping the bucket acquire for identical payloads within the window. This is a force multiplier for client-side rate limiting avoid 429 because it removes the call entirely.
if cached_response and not expired:
return cached_response
await bucket.acquire(1)Gateways like n4n.ai honor client routing directives and forward provider cache-control hints, so a local cache keyed on prompt hash compounds the savings without extra round trips.
Verify success
Success means: under sustained 2x expected load, your service logs zero 429s, p99 latency is stable, and the token bucket’s rate metric tracks the provider’s documented RPM within 5%. Re-run the load test weekly; providers tweak limits quietly.
That’s the whole loop. Ship the bucket, wrap the client, back off with jitter, and measure.