Debugging 500 vs 503 errors from LLM providers

When your LLM integration starts throwing HTTP errors, the difference between a 500 and a 503 from a provider dictates whether you should retry, fail over, or page a human. Misclassifying a transient 500 vs 503 error llm provider response leads to either needless retries against a dead server or premature aborts during routine load spikes. This guide walks through a concrete debugging workflow you can implement today, with runnable code and verification steps.

Step 1: Capture the exact status, headers, and body

Most high-level SDKs (openai, anthropic) catch non-2xx and raise a generic exception that hides the HTTP status. Before you can debug a 500 vs 503 error llm provider incident, you need the raw response. Wrap the SDK or drop to HTTP.

import requests

def raw_call(url, api_key, payload):
    resp = requests.post(
        url,
        headers={"Authorization": f"Bearer {api_key}"},
        json=payload,
        timeout=30,
    )
    return resp

resp = raw_call(
    "https://api.openai.com/v1/chat/completions",
    "sk-...",
    {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "ping"}]},
)
print("STATUS", resp.status_code)
print("RETRY-AFTER", resp.headers.get("retry-after"))
print("BODY", resp.text[:1000])

Capture retry-after even on 503; some providers send it, some don’t. Log the full body but redact any PII. Correlate with a request ID header if the provider supplies one (e.g., x-request-id). Without this data you are guessing.

Step 2: Classify the error by behavior, not just number

RFC 9110 says 500 is “unexpected condition prevented server from fulfilling”, 503 is “server not ready to handle request”, typically temporary. In practice, LLM providers blur the line. A 500 vs 503 error llm provider pair might both indicate GPU exhaustion depending on the backend.

Parse the JSON error object. Most OpenAI-compatible APIs return {"error": {"type": "...", "code": "...", "message": "..."}}.

import json

def extract_error(resp):
    try:
        data = resp.json()
    except json.JSONDecodeError:
        return {"type": "unknown", "code": str(resp.status_code), "message": resp.text}
    return data.get("error", {})

err = extract_error(resp)
print(err.get("type"), err.get("code"), err.get("message"))

Provider-specific patterns

If you see a 500 with a message containing “overload”, “capacity”, or “timeout”, treat it like a 503 for retry purposes. A plain 500 with “internal error” and no retry-after is suspect; retry once, then escalate.

Step 3: Implement a retry policy that respects semantics

Never blind-retry all 5xx. Write a predicate that allows 503 always, 429 always, and 500 only when the body suggests transient load. Use exponential backoff with full jitter to avoid thundering herds.

import requests
from tenacity import (
    retry, wait_exponential_jitter, stop_after_attempt,
    retry_if_exception_type,
)
import json

class ProviderError(Exception):
    def __init__(self, status, body):
        self.status = status
        self.body = body
        super().__init__(f"{status}: {body[:200]}")

def _is_retryable(exc):
    if not isinstance(exc, ProviderError):
        return False
    if exc.status in (429, 503):
        return True
    if exc.status == 500:
        try:
            msg = json.loads(exc.body).get("error", {}).get("message", "").lower()
        except Exception:
            msg = exc.body.lower()
        return any(k in msg for k in ("overload", "capacity", "timeout", "unavailable"))
    return False

@retry(
    retry=retry_if_exception_type(ProviderError) & retry_if_exception(_is_retryable),
    wait=wait_exponential_jitter(initial=1, max=15),
    stop=stop_after_attempt(5),
    reraise=True,
)
def call_llm(url, key, payload):
    r = requests.post(url, headers={"Authorization": f"Bearer {key}"}, json=payload, timeout=30)
    if r.status_code >= 500 or r.status_code == 429:
        raise ProviderError(r.status_code, r.text)
    r.raise_for_status()
    return r.json()

Set max based on user latency budget. For interactive apps, cap total retry time at 2–3 seconds. For batch, allow longer. Always emit a log line per retry with attempt number and status.

Step 4: Route around persistent provider failures

When a single provider returns a 503 storm, retrying the same endpoint wastes latency. A gateway that aggregates models across providers converts a provider-specific 503 into a cross-provider failover. n4n.ai provides one OpenAI-compatible endpoint covering 240+ models and performs automatic fallback when a provider is rate-limited or degraded, so your code sees a successful response instead of a 500 vs 503 error llm provider mismatch.

You can also pin routing directives:

requests.post(
    "https://api.n4n.ai/v1/chat/completions",
    headers={
        "Authorization": "Bearer N4NKEY",
        "x-n4n-route": "failover:anthropic,openai",
    },
    json={"model": "claude-3-5-sonnet", "messages": [{"role": "user", "content": "hi"}]},
)

If you run your own proxy, implement a circuit breaker (e.g., pybreaker) that opens after N consecutive 503s and routes to secondary.

Step 5: Instrument error rates per provider and status

You cannot debug what you don’t measure. Export counters partitioned by provider, model, and HTTP status. Use Prometheus or StatsD.

from prometheus_client import Counter

llm_errors = Counter(
    "llm_provider_http_errors_total",
    "Count of LLM provider HTTP errors",
    ["provider", "model", "status"],
)

def observe(provider, model, resp):
    if resp.status_code in (500, 503, 429):
        llm_errors.labels(provider=provider, model=model, status=resp.status_code).inc()

Build a dashboard that shows 500 vs 503 ratio side by side. A rising 503 line means capacity issue; a rising 500 line means code bug or provider incident. Alert when 503 rate exceeds 5% of requests over 5 minutes, or when 500 exceeds 1% sustained.

Also track retry success: how many initial 503s turned into 200 after backoff. This tells you if your retry policy is effective.

Step 6: Verify with fault injection

Theory is cheap. Simulate errors in CI and staging using responses (Python) or Toxiproxy for TCP-level chaos.

import responses
import pytest

@responses.activate
def test_retry_503_then_200():
    url = "https://api.openai.com/v1/chat/completions"
    responses.add(responses.POST, url, status=503, body='{"error":{"message":"overload"}}')
    responses.add(responses.POST, url, status=200, json={"choices": [{"message": {"content": "ok"}}]})
    result = call_llm(url, "key", {"model": "gpt-4o-mini", "messages": []})
    assert result["choices"][0]["message"]["content"] == "ok"
    assert len(responses.calls) == 2  # one fail, one success

@responses.activate
def test_no_retry_on_hard_500():
    url = "https://api.openai.com/v1/chat/completions"
    responses.add(responses.POST, url, status=500, body='{"error":{"message":"null pointer"}}')
    with pytest.raises(ProviderError):
        call_llm(url, "key", {"model": "gpt-4o-mini", "messages": []})
    assert len(responses.calls) == 1

Run these in your pipeline:

pytest tests/test_retry.py -v

For production verification, use a canary that forces a provider into maintenance mode (if your gateway supports it) and confirm users see no errors.

Verify success criteria

• CI tests pass for retry-on-503 and no-retry-on-hard-500.
• Dashboard shows zero user-visible 5xx after gateway failover during a simulated provider outage.
• Logs show retry attempts with decreasing frequency (jitter working).

Step 7: Document your error contract

Write a one-page internal note: which statuses are retryable, which trigger failover, which page on-call. Include the provider quirk table from Step 2. Share it with frontend and SRE teams. When the next 500 vs 503 error llm provider page fires at 3am, the on-call engineer should know within seconds whether to retry, flip traffic, or wait.

Update the contract quarterly; providers change behavior (OpenAI’s 500 semantics shifted after the November 2023 scale-up). Treat the error taxonomy as living code.