Most engineers treat an LLM gateway like a black box that returns text, but the contract is HTTP, and the llm api http status codes you get back determine whether your system survives a provider outage. A 200 with a malformed stream can hurt as much as a 500. This guide walks through the status codes that matter, what each implies about retrying, and how to code defensively.
2xx: success, but verify the shape
A 200 from a chat completion endpoint does not guarantee usable data. For streaming responses, the HTTP layer may succeed while the SSE stream terminates early. Always parse the final usage object or check for a closing done event before you declare victory.
async for chunk in stream:
if chunk.get("error"):
raise RuntimeError(chunk["error"])
if chunk.get("usage") is None and chunk.get("done") is True:
log.warning("stream ended without usage metering")If you rely on per-token metering, confirm the usage field is present. Some gateways omit it on timeout recovery, which breaks cost attribution downstream.
400 Bad Request: client-side contract violation
A 400 means your JSON failed schema validation, or you sent a parameter the model doesn’t support (e.g., temperature on a reasoning model). Do not retry blindly; log the response body and fix the call.
{
"error": {
"type": "invalid_request_error",
"message": "temperature is not supported for model xyz"
}
}Pitfall: some gateways return 422 Unprocessable Entity instead of 400. Treat them identically in your error handler.
401 and 403: credentials and scopes
401 means missing or malformed auth. 403 means the key is valid but lacks access to the requested model or route. Rotating keys fixes 401; 403 requires a permission change at the provider or a different routing directive.
Never implement retry-on-403 with the same token. That gets you flagged for abuse and can lock the key.
404: unknown model or bad route
A 404 on /v1/chat/completions with a valid path indicates the model ID does not exist or your routing directive pointed at a dead endpoint. If you pin a specific provider via header, a typo yields 404 even when the gateway itself is healthy.
curl -i -H "x-router: openai" https://gateway/v1/models/nonexistent
# HTTP/1.1 404 Not FoundTradeoff: generic model aliases (like gpt-4o) reduce 404 risk but may mask version drift. Pin exact IDs in production if reproducibility matters.
408 Request Timeout vs 429 Too Many Requests
408 is rare in LLM APIs; most gateways use 504 for upstream timeout. 429 is the workhorse of rate limiting. The llm api http status codes in the 4xx family that you must handle programmatically are dominated by 429.
429 Too Many Requests: back off, don’t hammer
A 429 carries a Retry-After header (seconds) or a reset timestamp in the body. Honor it. Exponential backoff without jitter causes thundering herds when many clients share a quota.
import asyncio, random
async def call_with_backoff(fn, max_retries=5):
for attempt in range(max_retries):
resp = await fn()
if resp.status != 429:
return resp
retry_after = resp.headers.get("Retry-After")
delay = float(retry_after) if retry_after else (2 ** attempt + random.random())
await asyncio.sleep(delay)
raise RuntimeError("exhausted retries on 429")Reading rate-limit headers
Most gateways send x-ratelimit-limit, x-ratelimit-remaining, and x-ratelimit-reset. Use remaining to shed load proactively before you hit 429.
{
"x-ratelimit-limit": "1000",
"x-ratelimit-remaining": "0",
"x-ratelimit-reset": "1700000000"
}Common pitfall: counting tokens locally and assuming you’re under limit. Provider limits include concurrent requests, not just TPM. A gateway may enforce its own quotas on top.
500, 502, 503, 504: upstream and gateway failures
Among llm api http status codes, 5xx are the only ones that safely permit blind retry with caps. 502/504 usually mean the model worker timed out or the upstream returned garbage. 503 means the service is temporarily unavailable.
Some gateways, including n4n.ai, perform automatic fallback when a provider is rate-limited or degraded, which converts many potential 503s into successful 200s; still, your client must handle the residual 503/504 from the gateway itself. Treat all 5xx as retryable with caps.
async function fetchWithRetry(url: string, opts: RequestInit, max = 3): Promise<Response> {
for (let i = 0; i < max; i++) {
const res = await fetch(url, opts);
if (res.status < 500) return res;
await new Promise(r => setTimeout(r, 200 * (i + 1)));
}
throw new Error("upstream 5xx persisted");
}When fallback hides the error
If the gateway retries upstream silently, your request may take 30s and then return 200 with partial data. Set a client timeout and validate completeness.
Tradeoff: blind retries on 500 can duplicate side effects if the request actually processed before crashing. For completions, idempotency keys help; pass a client-generated Idempotency-Key if the gateway supports it.
413 Payload Too Large: context window exceeded
If you send a 200k-token context to a 128k model, you’ll get 413 (or sometimes 400). This is not retryable with the same payload. Implement client-side truncation or use a model with a larger window.
Streaming errors after 200
A streaming response can fail mid-way: connection drops, or the provider raises an internal error serialized as an SSE error event. Your parser must detect it.
async def stream_chat():
async with client.stream("POST", "/v1/chat/completions") as r:
if r.status != 200:
raise RuntimeError(f"status {r.status}")
async for line in r.aiter_lines():
if line.startswith("data: "):
payload = json.loads(line[6:])
if "error" in payload:
raise RuntimeError(payload["error"])An actionable retry policy
Engineers need an ordered path, not a menu. Follow this:
- Classify status: 2xx (process), 400/401/403/404/413 (fail fast), 408/429 (backoff), 5xx (retry capped).
- Extract retry hints:
Retry-After,x-ratelimit-reset. - Use jitter and a max attempt count (usually 4–6).
- For streams, wrap parsing in try/except and treat parse failure as a 502.
- Emit metrics per status code; alert on 5xx rate, not just latency.
Why metrics matter
A spike in 429 tells you to scale quota or shed load; a spike in 503 tells you the gateway is unhealthy. Without per-code tagging, you only see “errors” and lose the signal.
Common pitfalls and tradeoffs
- Retrying POSTs without idempotency can double-bill usage. Gateways with per-token metering will count both attempts.
- Ignoring
Retry-Afterand using fixed sleep wastes quota and lengthens outages. - Treating 503 as fatal loses resilience; treating it as infinitely retryable causes stuck jobs.
- Client-side timeouts set too low generate 408-equivalent client errors but appear as connection resets, not clean llm api http status codes.
- Assuming 200 means success on streams: always check the final chunk for
erroror missingusage.
Reference client skeleton
import httpx, json, asyncio
class LLMClient:
def __init__(self, base_url, token):
self.client = httpx.AsyncClient(base_url=base_url,
headers={"Authorization": f"Bearer {token}"})
async def complete(self, payload, attempts=5):
for i in range(attempts):
r = await self.client.post("/v1/chat/completions", json=payload)
if r.status_code == 200:
return r.json()
if r.status_code in (400, 401, 403, 404, 413):
raise ValueError(f"fatal {r.status_code}: {r.text}")
if r.status_code == 429:
await asyncio.sleep(self._backoff(r, i))
continue
if r.status_code >= 500:
await asyncio.sleep(self._backoff(r, i))
continue
raise RuntimeError("max retries exceeded")
def _backoff(self, r, i):
ra = r.headers.get("Retry-After")
return float(ra) if ra else (2 ** i + 0.5)This skeleton separates fatal from transient codes and respects server hints. Wire it into your service and you’ll survive the next provider hiccup.