When your LLM client starts hitting HTTP 429 responses, the server usually tells you exactly when to retry via the Retry-After header. Respecting the retry-after header llm api rate limits is the difference between a resilient pipeline and a thundering herd that gets your API key throttled harder. This guide gives you an end-to-end implementation: parse the header, sleep correctly, bound your retries, and verify against a mock.
Step 1: Identify the 429 and extract the header
A well-behaved LLM provider returns 429 Too Many Requests with a Retry-After header when you exceed a quota. The header comes in two flavors:
- A bare integer: seconds to wait from the moment you received the response.
- An HTTP-date string: an absolute time after which the request may succeed.
HTTP/1.1 429 Too Many Requests
Retry-After: 12
Content-Type: application/json
{"error": {"type": "rate_limit_error"}}Some gateways omit the header entirely under certain degradations. Your client must cope with both cases.
Step 2: Parse both Retry-After formats
Do not assume the value is an integer. Use a parser that tries the delta form first, then falls back to date parsing. Python’s email.utils handles the HTTP-date format correctly.
from datetime import datetime, timezone
import email.utils
import time
def parse_retry_after(header_value, now=None):
if not header_value:
return None
now = now or datetime.now(timezone.utc)
try:
delta_sec = int(header_value)
return now.timestamp() + delta_sec
except ValueError:
pass
date = email.utils.parsedate_to_datetime(header_value)
if date.tzinfo is None:
date = date.replace(tzinfo=timezone.utc)
return date.timestamp()The function returns a Unix timestamp (absolute seconds) for when you are allowed to retry. If parsing fails, return None and let the caller apply a default backoff.
Step 3: Implement a first-pass retry loop
A naive recursive retry is easy to write and easy to get wrong. Below is a linear loop that respects the header and fails loudly after a bounded number of attempts.
import requests
def call_llm_basic(payload, api_key, base_url="https://api.openai.com/v1"):
url = f"{base_url}/chat/completions"
headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
resp = requests.post(url, headers=headers, json=payload)
if resp.status_code == 429:
wait = parse_retry_after(resp.headers.get("Retry-After")) - time.time()
if wait > 0:
time.sleep(wait)
return call_llm_basic(payload, api_key, base_url) # unbounded!
resp.raise_for_status()
return resp.json()This version has no retry cap and no jitter. In production, unbounded recursion will hang a worker indefinitely if the provider stays saturated.
Step 4: Add bounds, jitter, and fallback backoff
Wrap the logic in a small client class. When Retry-After is present, honor it. When it is missing, use exponential backoff. Always add jitter to avoid synchronized retries across many workers.
import random
class LLMClient:
def __init__(self, api_key, base_url="https://api.openai.com/v1", max_retries=5):
self.api_key = api_key
self.base_url = base_url
self.max_retries = max_retries
def complete(self, payload):
for attempt in range(self.max_retries):
resp = requests.post(
f"{self.base_url}/chat/completions",
headers={"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"},
json=payload,
)
if resp.status_code != 429:
resp.raise_for_status()
return resp.json()
wait = self._compute_wait(resp, attempt)
time.sleep(wait)
raise RuntimeError("Exhausted retries due to rate limits")
def _compute_wait(self, resp, attempt):
now = time.time()
raw = resp.headers.get("Retry-After")
if raw:
# respect retry-after header llm api rate limits from upstream
target = parse_retry_after(raw, datetime.now(timezone.utc))
base = max(0.0, target - now)
else:
base = (2 ** attempt) * 0.5 # 0.5s, 1s, 2s, 4s, 8s
# jitter up to 1s or the base wait, whichever is smaller
jitter = random.uniform(0, min(base, 1.0))
return base + jitterWhy jitter matters
If 1,000 workers all receive Retry-After: 10 and sleep exactly 10 seconds, they will fire simultaneously at second 10 and likely trigger another 429. Jitter spreads the load.
Clamping insane values
A buggy or malicious proxy could send Retry-After: 86400. Cap the maximum wait to something your task can tolerate, e.g. 60 seconds, then fail the request.
MAX_WAIT = 60.0
base = min(base, MAX_WAIT)Step 5: Handle missing or malformed headers
When the header is absent, exponential backoff is the standard fallback. When the header is present but unparsable, treat it as absent rather than crashing. Log the raw value for debugging.
raw = resp.headers.get("Retry-After")
if raw:
try:
target = parse_retry_after(raw, datetime.now(timezone.utc))
base = max(0.0, target - time.time())
except Exception:
base = None
if base is None:
base = (2 ** attempt) * 0.5Negative waits (clock skew) should be clamped to zero so you retry immediately rather than sleeping backward.
Step 6: Stand up a mock server to verify behavior
You cannot verify backoff logic against a live paid API without spending money and risking throttling. Use Flask to simulate a provider that rate-limits the first call, then succeeds.
from flask import Flask, jsonify, request
app = Flask(__name__)
state = {"calls": 0}
@app.route("/v1/chat/completions", methods=["POST"])
def completions():
state["calls"] += 1
if state["calls"] == 1:
return jsonify({"error": "rate limited"}), 429, {"Retry-After": "2"}
return jsonify({"choices": [{"message": {"content": "hello"}}]})
if __name__ == "__main__":
app.run(port=5000)Run the mock, then point the client at it:
client = LLMClient("fake-key", base_url="http://localhost:5000")
start = time.time()
result = client.complete({"model": "gpt-4o-mini", "messages": []})
print(f"Got response after {time.time() - start:.1f}s: {result}")You should see the call block for roughly 2 seconds (plus jitter) and then return the success payload.
Step 7: Production hardening and gateway fallback
In async environments, replace time.sleep with await asyncio.sleep and use an HTTP client like aiohttp or httpx. Reuse a connection pool to avoid TCP handshake overhead on every retry.
If you sit behind an OpenAI-compatible inference gateway, the 429 behavior may differ. For example, n4n.ai provides automatic fallback when a provider is rate-limited or degraded, which means many requests never surface a 429 to your code. When a Retry-After is still returned, honoring the retry-after header llm api rate limits prevents you from hammering the fallback target while it recovers. The same parsing code works unchanged because the gateway forwards standard HTTP semantics.
Also consider per-token metering and client routing directives: if your gateway supports them, set explicit route hints so that retries land on a different provider shard rather than the same saturated one.
How to verify success
- Unit test the parser: feed
"30"and"Wed, 31 Dec 2025 23:59:59 GMT"toparse_retry_afterand assert the returned timestamps. - Integration test with mock: run the Flask server above and confirm the client succeeds on the second attempt after ~2s.
- Chaos test: remove the
Retry-Afterheader from the mock and confirm exponential backoff kicks in (add logging to show waits). - Live smoke test: against a real API, intentionally send a burst of requests, observe 429s, and check that your logs show waits matching the header values.
Respecting rate limits is not optional for stable LLM pipelines. The Retry-After header is a contract; parse it strictly, bound your retries, and verify the behavior before you ship.