Most LLM integrations fail not because the model is wrong, but because a provider throttles requests at the worst moment. A solid retry queue design llm api rate limits separates transient failures from real ones, buffers load, and lets your service degrade gracefully instead of melting down.
1. Map the failure modes before writing code
Any retry queue design llm api rate limits must start with ruthless error classification. HTTP 429 is the obvious signal, but LLM APIs also return 500/502/503, drop connections, or hang past your client timeout. Treat only a subset as retryable.
Classify errors at the boundary:
429,500,502,503,504→ retryable- Connection reset, TLS error, socket timeout → retryable
400,401,403→ non-retryable (fix the request, not the schedule)422with validation error → non-retryable
A common pitfall is catching every Exception and requeueing. You’ll loop on bad auth forever and flood your own queue. Log the status code and map it to a policy at the HTTP client layer, not inside the worker logic.
Distinguish provider quotas from global limits
Some providers rate-limit per model, per organization, or per token bucket. A 429 with error.type: requests means too many concurrent calls; error.type: tokens means you blew the daily spend. Your queue should back off longer on token-limit errors because they rarely clear in seconds.
2. Pick a queue that matches your durability needs
For a single-process worker handling short bursts, an in-memory asyncio.Queue is enough. It’s fast and needs no infra, but a crash loses everything pending.
If you run multiple workers or need survival across deploys, use Redis or Postgres. A minimal Redis job:
{
"id": "job_8f3a",
"payload": {"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]},
"attempts": 0,
"max_attempts": 5,
"next_run": 1715000000,
"idempotency_key": "req_123",
"priority": 1
}Enqueue with RPUSH, consume with BLPOP or a sorted set keyed by next_run for delayed retries. The tradeoff is operational overhead: you now own a datastore, backups, and monitoring for the queue itself.
In-memory sketch
import asyncio
queue = asyncio.Queue(maxsize=1000)
async def producer(req):
await queue.put(req)
async def consumer():
while True:
job = await queue.get()
await process(job)
queue.task_done()Durable sketch
# enqueue
redis-cli RPUSH llm_jobs '{"id":"job_1","attempts":0,"next_run":0}'
# delayed pop (poll)
redis-cli ZRANGEBYSCORE llm_delayed 0 $(date +%s) LIMIT 0 10A durable queue forces you to handle partially processed jobs after a worker kill. Use next_run as a score in a ZSET so delayed retries don’t block the active list.
3. Apply exponential backoff with jitter and caps
Backoff is core to retry queue design llm api rate limits. Naive fixed-delay retries cause synchronized retry storms when a provider recovers. Use exponential backoff with full jitter:
import random, time
def backoff_delay(attempt: int, base: float = 1.0, cap: float = 60.0) -> float:
delay = min(cap, base * (2 ** attempt))
return random.uniform(0, delay)
# inside worker
time.sleep(backoff_delay(job["attempts"]))The attempt counter starts at 0. After attempt 3, delay is capped at 60s. Jitter spreads the load so thousands of queued jobs don’t hammer the API on the same tick.
Why full jitter beats equal jitter
Equal jitter (delay/2 + random(delay/2)) still clusters near the mean. Full jitter (random(0, delay)) maximizes spread. For LLM providers where recovery is sudden, full jitter prevents a secondary outage caused by your own clients.
A subtle bug: using sleep in an async worker blocks the event loop. Await asyncio.sleep instead, or run blocking sleeps in a thread pool.
4. Make jobs idempotent and track attempts
LLM calls often have side effects (storing a generated row). If a request times out after the provider actually processed it, a blind retry duplicates work. Require an idempotency_key and have downstream consumers dedupe.
def execute(job):
key = job["idempotency_key"]
if cache.get(key):
return cache.get(key) # already done
result = llm_client.create(**job["payload"])
cache.set(key, result, ttl=86400)
return resultTrack attempts in the job. When attempts >= max_attempts, route to a dead-letter queue (DLQ) instead of looping. This caps your worst-case resource burn.
Storing idempotency keys
Use a TTL longer than your max retry window plus any user-visible lag. Redis or a hashed database column works. Don’t rely on the provider’s own idempotency support alone—your queue may retry against a different provider after failover.
5. Respect rate limits with a local limiter
Even with retries, you should avoid hitting limits in the first place. A token-bucket limiter in the worker process smooths outgoing calls:
import asyncio
class Limiter:
def __init__(self, rate: int, capacity: int):
self.rate = rate
self.capacity = capacity
self.tokens = capacity
self.updated = asyncio.get_event_loop().time()
self.lock = asyncio.Lock()
async def acquire(self):
async with self.lock:
now = asyncio.get_event_loop().time()
self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
self.updated = now
if self.tokens < 1:
await asyncio.sleep((1 - self.tokens) / self.rate)
self.tokens = 1
self.tokens -= 1Rate limiters complement retry queue design llm api rate limits by reducing the 429 rate before it happens. Pair this with priority: paid-tier requests jump the queue, batch jobs wait.
Priority queue implementation
Use heapq with a tuple (priority, next_run, id). Lower priority number runs first. This prevents low-value backfills from starving interactive traffic during throttling.
6. Add observability and a dead-letter queue
You can’t operate what you can’t see. Export metrics: queue depth, jobs per second, retry rate, DLQ size.
from prometheus_client import Counter, Gauge
QUEUE_DEPTH = Gauge('llm_queue_depth', 'Pending jobs')
RETRIES = Counter('llm_retries_total', 'Retry attempts')
DLQ = Counter('llm_dlq_total', 'Dead-lettered jobs')When a job exhausts attempts, push to a separate Redis list dlq: and alert. A DLQ lets you inspect failed payloads without blocking live traffic. The tradeoff: someone must actually look at it, or it becomes a graveyard.
Alerting thresholds
Page if DLQ growth exceeds 1% of total throughput for five minutes. Warn if queue depth stays above 10x worker capacity for ten minutes—that signals a provider outage or a misconfigured limiter.
7. Offload multi-provider failover to a gateway
If you sit behind a gateway like n4n.ai, which provides automatic fallback when a provider is rate-limited or degraded, your retry queue can skip explicit multi-provider logic and just retry the same OpenAI-compatible request. The gateway honors your routing hints and forwards cache-control, so a retry that hits a different provider still benefits from prompt caching where supported.
This simplifies your queue: you only handle generic transient errors, not provider-specific quota maps. Keep the queue for buffering and backoff; let the gateway handle provider selection.
8. Test the queue with fault injection
A retry queue that hasn’t been tested against chaos is a guess. Use Toxiproxy or a local mock that returns 429 for 30% of calls. Verify:
- Jobs delay according to backoff, not instantly.
- DLQ receives jobs after
max_attempts. - Idempotency cache prevents duplicate side effects.
- Queue depth metric moves correctly.
Run a game day where you block the primary provider entirely. Confirm the gateway fallback engages and your queue drain rate recovers within the backoff cap.
Common pitfalls and tradeoffs
Retry storms after outage. Without jitter and caps, a recovered provider gets slammed. Always jitter.
Coupling retries to user latency. Don’t make the end user wait for five backoff attempts. Return a 202 with a job ID and poll, or use websockets.
Ignoring clock skew. If next_run uses server time and you deploy across zones, delayed jobs fire early or late. Use UTC and NTP.
Unbounded queue growth. A durable queue with a slow consumer fills disk. Add max depth and reject new jobs with 503 when full.
Missing timeouts on the client. An LLM call without a timeout hangs the worker. Set request_timeout and treat timeout as a retryable failure.
Over-retrying non-idempotent writes. If your job inserts to a database before calling the model, a retry may double-insert. Design jobs as pure functions of their idempotency key.
Assuming one retry policy fits all models. A small embedding model can be retried aggressively; a 100k-token completion cannot. Parameterize base delay and cap per model class.
The retry queue design llm api rate limits problem is mostly about disciplined classification, bounded delays, and separating transient from permanent failure. Build the smallest queue that meets your durability needs, measure it under fault injection, and let a gateway handle cross-provider routing if you can.