Timeout and connection error handling for LLM REST calls

Calling an LLM over HTTP is deceptively easy until a provider hangs the socket. Robust python timeout error handling llm rest api code separates flakes from real failures, keeps your service responsive, and avoids burning tokens on duplicate requests. This guide walks an ordered path from explicit timeouts to retry policies and async patterns you can ship today.

1. Set explicit timeouts on every request

The fastest way to take down your own service is to let an outbound LLM call block forever. In requests, the default timeout is None, meaning a stalled connection never returns. httpx applies a default 5-second timeout, but relying on a library default is lazy and breaks when your model needs 40 seconds to generate.

Always pass an explicit timeout. For requests, use a (connect, read) tuple:

import requests

resp = requests.post(
    "https://api.example.com/v1/chat/completions",
    json={"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]},
    timeout=(3.05, 30)  # connect budget 3.05s, read budget 30s
)

For httpx, construct a Timeout object so you can also bound write and pool waits:

import httpx

timeout = httpx.Timeout(connect=3.05, read=30.0, write=5.0, pool=5.0)
resp = httpx.post(
    "https://api.example.com/v1/chat/completions",
    json={"model": "gpt-4o", "messages": [{"role": "user", "content": "hi"}]},
    timeout=timeout
)

The connect budget covers DNS resolution and TCP/TLS handshake. The read budget covers time between bytes from the server—not total response time for streaming. For non-streaming completions, set read to your worst-case generation latency plus headroom.

2. Catch the right exceptions

A generic except Exception hides programming errors and makes retries dangerous. Both libraries expose a precise hierarchy.

With requests:

from requests.exceptions import Timeout, ConnectionError, HTTPError

try:
    resp = requests.post(url, json=payload, timeout=(3.05, 30))
    resp.raise_for_status()
except Timeout as e:
    # ConnectTimeout or ReadTimeout; safe to retry
    log("timeout", str(e))
except ConnectionError as e:
    # DNS failure, refused connection, reset peer
    log("connerror", str(e))
except HTTPError as e:
    # 4xx/5xx already raised by raise_for_status()
    log("httperror", e.response.status_code)

With httpx (sync or async, same class names):

import httpx

try:
    r = httpx.post(url, json=payload, timeout=timeout)
    r.raise_for_status()
except httpx.ReadTimeout:
    log("read_timeout")
except httpx.ConnectTimeout:
    log("connect_timeout")
except httpx.ConnectError:
    log("connect_error")
except httpx.HTTPStatusError as e:
    log("status_error", e.response.status_code)

Do not conflate httpx.TransportError with retryable conditions—it includes some non-retryable SSL failures. Catch the narrow subclasses.

3. Retry only what is safe

Network flakes and provider overload deserve retries; client errors do not. Use tenacity to express the policy declaratively:

from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
import requests
from requests.exceptions import Timeout, ConnectionError, HTTPError

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=1, max=10),
    retry=retry_if_exception_type((Timeout, ConnectionError)),
    reraise=True,
)
def call_llm(payload):
    r = requests.post(url, json=payload, timeout=(3.05, 30))
    # Retry on 429 and 5xx but not 4xx
    if r.status_code == 429 or r.status_code >= 500:
        r.raise_for_status()  # raises HTTPError, but we need to retry that too
    r.raise_for_status()
    return r.json()

Extend the retry predicate to include HTTPError with a status check:

from tenacity import retry_if_exception

def is_retryable(e):
    if isinstance(e, (Timeout, ConnectionError)):
        return True
    if isinstance(e, HTTPError) and e.response is not None:
        return e.response.status_code == 429 or e.response.status_code >= 500
    return False

@retry(retry=retry_if_exception(is_retryable), stop=stop_after_attempt(3), wait=wait_exponential(1,1,10), reraise=True)
def call_llm_v2(payload):
    ...

Tradeoff: retrying a generation call can produce a different completion unless you pin temperature=0 and pass a seed where the API supports it. If you front calls with a gateway that provides automatic fallback when a provider is rate-limited or degraded, your client still owns transport-level timeouts and must decide whether to retry a failed TCP connection.

4. Separate connect and read budgets

Engineers often set a single scalar timeout and wonder why short network blips kill long generations. Split the budget:

• Connect: 2–5 seconds. If you cannot establish a TLS session in that window, the provider edge is unhealthy; fail fast.
• Read: 30–90 seconds depending on max_tokens and model. For streaming, read is inactivity timeout—each chunk resets it.

httpx also exposes pool timeout: time waiting for a connection from the pool. If you see httpx.PoolTimeout, you have pool exhaustion, not a slow model.

5. Async clients and cancellation

In an async service, a hung LLM call holds an event loop slot. Use httpx.AsyncClient with explicit timeouts and wrap with asyncio.wait_for as a last-resort guard:

import asyncio, httpx

async def call_llm_async(payload):
    timeout = httpx.Timeout(connect=3.05, read=60.0)
    async with httpx.AsyncClient(timeout=timeout) as client:
        try:
            resp = await client.post(url, json=payload)
            resp.raise_for_status()
            return resp.json()
        except httpx.ReadTimeout:
            # treat as retryable at caller
            raise

# Outer guard if you want hard cap:
async def bounded_call(payload):
    return await asyncio.wait_for(call_llm_async(payload), timeout=75.0)

If you cancel the task, close the client to avoid leaking sockets. Never share a single AsyncClient across unrelated loop lifetimes without cleanup.

6. Connection pooling and limits

Reusing connections matters: TLS handshakes to LLM endpoints add 100–300 ms. Use a session:

import requests

session = requests.Session()
session.mount("https://", requests.adapters.HTTPAdapter(pool_connections=20, pool_maxsize=20))

# reuse session for all calls
session.post(url, json=payload, timeout=(3.05, 30))

httpx equivalent:

client = httpx.Client(limits=httpx.Limits(max_connections=100, max_keepalive_connections=20))

Pitfall: setting max_connections too low under bursty traffic causes PoolTimeout. Too high and you can overwhelm your own network or hit client-side file descriptor limits.

7. Logging and metrics

Capture enough to debug without leaking prompts. Structured log lines:

log({
    "event": "llm_call_failed",
    "error_type": type(e).__name__,
    "url": url,
    "connect_timeout": 3.05,
    "read_timeout": 30,
    "attempt": attempt,
    "elapsed_ms": int(e.elapsed.total_seconds()*1000) if hasattr(e,'elapsed') else None
})

Emit a counter per error class. A sudden spike in ConnectError means provider edge issue; a spike in ReadTimeout means model saturation.

8. Common pitfalls

• No timeout in requests: The default None will hang your worker indefinitely.
• Scalar timeout too small: 5-second read on a 70-token/second model generating 512 tokens guarantees timeouts.
• Retrying 400 errors: Wastes calls and tokens; only 429/5xx are retryable.
• Ignoring Retry-After: Respect the header on 429; exponential backoff without it is rude.
• Catching Exception: Masks KeyboardInterrupt, SystemExit, and code bugs.
• Sync client in async loop: Blocks the loop; use httpx.AsyncClient or run sync in executor.
• SSL verification disabled: verify=False hides cert rotation outages and exposes MITM.

9. Ordered implementation checklist

1. Create a single Session/Client with pooled connections and explicit (connect, read) timeouts.
2. Wrap calls in a function that catches Timeout, ConnectionError, and status-based HTTPError separately.
3. Apply a retry decorator limited to connection errors, timeouts, 429, and 5xx with exponential backoff and Retry-After honor.
4. For async, use AsyncClient with same timeout split and an outer asyncio.wait_for only if you need a hard process cap.
5. Log error class, elapsed, and attempt; emit metrics.
6. Pin temperature=0 and seed when deterministic output across retries matters.
7. Load-test with injected latency to confirm your read budget holds under real generation times.

Follow that path and your python timeout error handling llm rest api layer will survive provider flakiness without taking your application down with it.