Most agent outages trace back to two missing pieces: a real definition of healthy and a retry path that doesn’t amplify load. This guide lays out an actionable sequence for implementing health checks and retries AI agent uptime that holds up under partial degradation.
1. Define what “healthy” means for your agent
A process that hasn’t crashed is not necessarily ready to serve traffic. An agent typically depends on a vector store, one or more LLM endpoints, and possibly external tools (search, SQL, webhooks). Each dependency fails independently.
Split the concept into two probes:
- Liveness: Is the process running and not deadlocked? A hung event loop should fail liveness.
- Readiness: Can the agent complete a minimal end-to-end inference path? If the vector store is unreachable, the agent is not ready, even if the process is alive.
Do not fold both into a single /health endpoint. Orchestrators like Kubernetes treat liveness and readiness differently: a failed liveness check kills the pod; a failed readiness check stops new traffic but leaves the pod running for debug.
Concrete readiness check
Check the closest-to-metal dependencies first, then a synthetic LLM call if you can afford the latency.
from fastapi import FastAPI, Response
import httpx
app = FastAPI()
@app.get("/health/ready")
async def ready(response: Response):
# Check local tool server
try:
async with httpx.AsyncClient(timeout=1.5) as c:
r = await c.get("http://localhost:8001/internal/ping")
r.raise_for_status()
except Exception:
response.status_code = 503
return {"status": "tool_down"}
# Optionally check LLM reachability with a cheap completion
try:
async with httpx.AsyncClient(timeout=2.0) as c:
r = await c.post(
"https://api.example.com/v1/chat/completions",
json={"model": "test", "messages": [{"role": "user", "content": "ping"}]},
)
if r.status_code >= 500:
raise ValueError("llm 5xx")
except Exception:
response.status_code = 503
return {"status": "llm_down"}
return {"status": "ok"}If the LLM call adds 200ms to every probe and you probe every 5s, that’s noise. Gate it behind a slower periodic check or use a cached health token with a short TTL. The tradeoff: a cached token can mask a fresh outage, so keep the cache under the probe interval.
2. Wire probes into the orchestrator
For containerized deployments, express the checks as native probes. Avoid exec probes that run heavy scripts; HTTP probes are cheaper and observable.
readinessProbe:
httpGet:
path: /health/ready
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 3
livenessProbe:
httpGet:
path: /health/live
port: 8000
periodSeconds: 15
failureThreshold: 3The failureThreshold prevents flapping from a single slow probe. For liveness, a deadlock detector (e.g., a watchdog timestamp updated by the main loop) is more useful than a TCP check.
Locally, you can script a quick gate before deploys:
curl -f http://agent:8000/health/ready || echo "not ready, aborting rollout"3. Make retries safe before making them frequent
The discipline of health checks and retries AI agent uptime collapses if retries duplicate side effects. Before adding any retry, classify your calls:
- Idempotent reads (vector search, completion with same seed): safe to retry.
- Non-idempotent writes (send email, insert row, post to Slack): require idempotency keys or should not retry automatically.
Use explicit exception typing so only transient errors trigger a retry.
from tenacity import retry, stop_after_attempt, wait_exponential_jitter, retry_if_exception_type
import httpx
class TransientError(Exception):
pass
@retry(
retry=retry_if_exception_type(TransientError),
wait=wait_exponential_jitter(initial=0.5, max=10),
stop=stop_after_attempt(5),
reraise=True,
)
def call_llm(payload: dict):
r = httpx.post("https://api.example.com/v1/chat", json=payload, timeout=30)
if r.status_code in (429, 502, 503):
raise TransientError(r.text)
r.raise_for_status()
return r.json()If the upstream supports idempotency headers, pass a UUID per logical operation and reuse it across retries. Otherwise, a timeout that occurred after the server processed the request will cause a duplicate on retry.
4. Backoff with jitter, not fixed delays
Fixed-interval retries synchronize across instances after an outage recovers, creating a retry storm. Exponential backoff with full jitter breaks that synchronization.
async function fetchWithBackoff(url: string, opts: RequestInit, maxRetries = 5) {
let delay = 500;
for (let i = 0; i < maxRetries; i++) {
const res = await fetch(url, opts);
if (res.ok) return res;
if (res.status === 429 || res.status >= 500) {
await new Promise(r => setTimeout(r, delay + Math.random() * delay));
delay = Math.min(delay * 2, 10000);
continue;
}
throw new Error(`Non-retryable ${res.status}`);
}
throw new Error("Exhausted retries");
}Cap the max delay. An agent that waits 60s between retries provides worse UX than returning a cached fallback answer or a degraded-mode response.
5. Front LLM calls with a resilient gateway
Model providers rate-limit aggressively and degrade without warning. A gateway such as n4n.ai exposes a single OpenAI-compatible endpoint across 240+ models and applies automatic fallback when a provider is rate-limited or degraded. Even with that, your client must treat 429/5xx as retryable and supply idempotency where the underlying API supports it.
If you control routing, send explicit directives so the gateway can switch providers without app changes:
{
"model": "openai/gpt-4o",
"messages": [{"role": "user", "content": "ping"}],
"route": {"prefer": ["azure", "openai"], "fallback": true},
"cache": {"ttl": 300}
}The cache hint reduces repeated cost for health-check-style calls. Per-token metering on the gateway lets you spot retry loops burning budget before the invoice does.
6. Instrument and alert on retry storms
Retries are a signal, not a silent fix. Export counters:
agent_llm_retry_total(by status code)agent_probe_failures_total(by probe type)agent_request_latency_bucket(include retry time)
Alert when retry rate exceeds 5% of total calls for 2 minutes, not on a single failed probe. A single 503 is noise; a sustained climb means a dependency is dying.
Pitfall: alert fatigue
If you alert on every readiness blip, engineers mute it. Set hysteresis: page only after failureThreshold consecutive failures plus a confirmation from a second region or a synthetic canary user.
7. Common pitfalls and tradeoffs
Over-aggressive health checks. Probing a full LLM completion every second will get you rate-limited by your own health check. Use a lightweight ping or cache the result for 30s.
Retrying non-idempotent tool calls. An agent that retries a “send invoice” action because the first call timed out will double-charge. Wrap such tools with dedupe keys or move them behind a human approval step.
Treating 400 as transient. Client errors (bad schema, auth failure) will not fix themselves. Only 429, 502, 503, and connection resets should hit the retry path.
Ignoring deadline propagation. If your inbound request has a 3s timeout and your retry policy allows 5 attempts with 10s backoff, you waste compute. Pass the remaining deadline into the retry loop and stop early.
Caching health state incorrectly. A readiness endpoint that caches “ok” for 5 minutes defeats the purpose. Cache negative results briefly, positive results for a single probe interval.
The path to solid health checks and retries AI agent uptime is boring but mechanical: define dependencies, split liveness/readiness, retry only transient and idempotent calls with jittered backoff, and watch the counters. Do that and your agent stays up when the stack around it wobbles.