A Django service that calls an LLM provider lives or dies by how it handles the network. If you treat django llm api timeouts retries as an afterthought, you will bleed requests during provider hiccups and confuse users with duplicate generations. This guide lays out a concrete order of operations for making those calls resilient without turning your views into spaghetti.
1. Set explicit timeouts at the HTTP client level
Django does not enforce outbound HTTP timeouts by default. If you use requests without a timeout argument, a stalled connection can hang a worker indefinitely. Start by pinning both connect and read timeouts on every LLM call.
import httpx
from django.conf import settings
def call_llm(payload: dict) -> dict:
with httpx.Client(timeout=httpx.Timeout(
connect=5.0,
read=30.0,
write=5.0,
pool=5.0,
)) as client:
resp = client.post(
settings.LLM_ENDPOINT,
json=payload,
headers={"Authorization": f"Bearer {settings.LLM_KEY}"},
)
resp.raise_for_status()
return resp.json()A 30-second read timeout is generous for most chat completions but tight enough to fail fast under degradation. Tune it based on your model’s p95 latency, not its marketing sheet.
2. Separate connect and read timeouts
The foundation of django llm api timeouts retries is recognizing that a slow TCP handshake and a slow token stream are different failures. Connect timeouts should be short (2–5s) because DNS or TLS issues rarely self-heal. Read timeouts can be longer because the model is actually computing.
timeout = httpx.Timeout(connect=3.0, read=45.0, write=5.0, pool=2.0)If you bundle them into one scalar, you either retry too aggressively on slow generation or tolerate dead connections too long.
3. Use a retry library with exponential backoff
Hand-rolled for loops with time.sleep are a maintenance tax. Use tenacity to apply bounded exponential backoff and only retry on transient errors.
from tenacity import (
retry, stop_after_attempt, wait_exponential,
retry_if_exception_type, reraise
)
import httpx
@retry(
stop=stop_after_attempt(4),
wait=wait_exponential(multiplier=1, max=10),
retry=retry_if_exception_type((httpx.TimeoutException, httpx.HTTPStatusError)),
reraise=True,
)
def call_llm_with_retry(payload: dict, idem_key: str) -> dict:
with httpx.Client(timeout=httpx.Timeout(connect=3.0, read=45.0)) as client:
resp = client.post(
settings.LLM_ENDPOINT,
json=payload,
headers={
"Authorization": f"Bearer {settings.LLM_KEY}",
"Idempotency-Key": idem_key,
},
)
# Only retry on 429, 502, 503, 504
if resp.status_code in (429, 502, 503, 504):
resp.raise_for_status()
return resp.json()Note the explicit status filter. Retrying on 400 wastes quota and corrupts user input.
4. Make retries safe with idempotency keys
When designing django llm api timeouts retries, assume that a timeout does not mean the server didn’t process your request. If the provider supports an Idempotency-Key header (OpenAI-compatible APIs do), generate one per logical user action and reuse it across retries.
import uuid
def handle_chat(request):
idem_key = request.headers.get("X-Idempotency-Key") or str(uuid.uuid4())
payload = {"messages": [...], "model": "gpt-4o"}
try:
return call_llm_with_retry(payload, idem_key)
except Exception:
# surface to user or queueWithout this, a retried POST can double-bill tokens or produce two side effects in a RAG pipeline.
5. Add a circuit breaker for repeated failures
Retries alone cause retry storms when a provider is hard-down. A circuit breaker blocks calls after fail_max consecutive errors and lets the dependency recover.
import pybreaker
llm_breaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=60)
@llm_breaker
def call_llm_protected(payload, idem_key):
return call_llm_with_retry(payload, idem_key)Tradeoff: a half-open breaker will let one request through to probe health. Ensure your monitoring distinguishes breaker-open errors from real timeouts so you don’t page on the wrong signal.
6. Offload long calls to background workers
Django’s request/response cycle is not the place for a 40-second LLM generation. Push the call into Celery (or Django-Q) and poll or stream the result.
from celery import shared_task
@shared_task(bind=True, max_retries=3, default_retry_delay=5)
def generate_llm_task(self, payload, idem_key):
try:
return call_llm_protected(payload, idem_key)
except pybreaker.CircuitBreakerError as exc:
# don't retry, fail fast to caller
raise self.retry(exc=exc, countdown=30, max_retries=0)This decouples user-facing latency from provider latency and lets you apply django llm api timeouts retries inside the worker where a crash is cheap.
7. Handle streaming and partial responses
If you stream tokens via SSE, a dropped connection mid-stream is not a full failure. Capture what you got, and on reconnect, request continuation with the prior Idempotency-Key and a stream_offset if the API supports it. Otherwise, fall back to non-streaming for the retry to get a complete JSON object.
# Pseudocode for stream resume
def stream_with_fallback(payload, idem_key):
try:
for chunk in stream_llm(payload, idem_key):
yield chunk
except httpx.ReadTimeout:
# final attempt: full blocking call
yield call_llm_protected(payload, idem_key)["choices"][0]["message"]8. Instrument everything
You cannot tune timeouts blind. Emit structured logs with attempt number, status, and latency per call. Use Django’s logging or OpenTelemetry spans.
import logging
logger = logging.getLogger("llm")
def call_llm_with_retry(payload, idem_key):
for attempt in range(4):
try:
# ... http call
logger.info("llm_ok", extra={"attempt": attempt, "model": payload["model"]})
return resp.json()
except httpx.TimeoutException:
logger.warning("llm_timeout", extra={"attempt": attempt})
# tenacity handles sleepPer-token metering (if your provider or gateway exposes it) tells you whether retries are costing real money, not just latency.
9. Consider a gateway that absorbs fallback
If you front your providers with an OpenAI-compatible gateway (n4n.ai exposes one endpoint covering 240+ models with automatic fallback when a provider is rate-limited or degraded), you can shed some client retry burden. The gateway retries across backends so your Django code sees fewer 503s. You still must set client timeouts and idempotency keys—fallback is not a substitute for failing fast on a dead socket.
Common pitfalls and tradeoffs
- Retrying on 4xx except 429. A
400means your payload is wrong. Retrying it burns attempts and obscures bugs. - Timeout values copied from tutorials. A 10s read timeout might be fine for embeddings but lethal for long-form generation. Measure your p95.
- Synchronous retries in a view. Blocking the worker multiplies capacity loss during incidents. Move to tasks.
- No jitter. Plain exponential backoff synchronizes clients. Add
wait_exponential_jitterfrom tenacity to spread load. - Ignoring connection pooling.
httpx.Clientas a context manager per call defeats pooling. Instantiate a module-level client withlimits=httpx.Limits(max_connections=100).
Most django llm api timeouts retries implementations stop at a @retry decorator and call it done. The resilient pattern is layered: client timeouts → filtered backoff → idempotency → circuit breaker → worker isolation → gateway fallback. Build it in that order, and your Django app will survive the next provider outage without waking you at 3am.