Gateway failure detection latency determines how many user requests hit a dead model backend before traffic reroutes. In a multi-provider LLM gateway, this latency is not a single number but a function of detection method, health-check cadence, and circuit-breaker state. Engineers sizing a failover system need to separate hard failures (connection refused, timeout) from soft degradation (high error rate, slow tokens) because the gateway detects them on different timescales, and the difference drives your user-visible outage window.
The anatomy of a failure
A provider behind an inference gateway can fail in three distinct ways:
- Hard down – TCP connection refused, DNS failure, or TLS handshake error. The gateway knows immediately at connection time, typically within 1–5 ms on a warm socket. No request body is sent.
- Timeout – The connection opens but no bytes return within the client timeout. Detection waits for the timeout to fire, usually 500 ms–10 s depending on configuration.
- Degraded – The provider accepts requests but returns 429/5xx or streams tokens at a fraction of normal speed. The gateway only notices after enough requests sample the bad behavior.
Gateway failure detection latency is dominated by the second and third cases. The first is trivial; the interesting engineering is minimizing exposure to the others without penalizing healthy-but-slow providers.
LLM workloads add a fourth wrinkle: streaming stalls. A provider may send the first token, then hang. If your gateway only watches HTTP status, it reports success while the user stares at a frozen cursor. Detecting that requires either an inter-token timeout or a passive observation of throughput, which pushes detection latency into the seconds range.
Passive detection: what the request tells you
The fastest signal is the one you already paid for: the request itself. If a call to a provider raises an exception, the gateway can mark that provider unhealthy synchronously. No extra network round-trip required.
def route_with_passive_detection(request, providers):
for provider in providers.ordered():
if circuit_breaker[provider].is_open():
continue
try:
return provider.send(request, timeout=0.8)
except (ConnectionError, TimeoutError) as e:
circuit_breaker[provider].record_failure()
# gateway failure detection latency here is ~0 ms after the error returns
raise AllProvidersDown()Passive detection adds zero separate probing overhead. Its downside is that the first user request to a dead provider eats the full timeout. If your timeout is 2 seconds, that user waits 2 seconds before fallback. That is unacceptable for interactive LLM apps where 95th-percentile latency budgets are often under 1.5 s.
Thus passive detection alone sets a worst-case gateway failure detection latency equal to your request timeout. You can shrink the timeout, but too aggressive a value false-trips on slow-but-alive providers (common during provider load spikes). For streaming, you need a secondary timeout: if no token arrives in first_token_ms or inter-token gap exceeds stall_ms, treat it as failure.
Active health checks: trading overhead for foresight
Active checks decouple detection from user traffic. The gateway periodically pings each provider’s status endpoint or sends a tiny completion. When the check fails, the circuit opens before the next user request.
{
"provider": "openai",
"health_check": {
"interval_ms": 2000,
"timeout_ms": 400,
"path": "/v1/models",
"failure_threshold": 3
}
}With a 2 s interval and three consecutive failures required, the worst-case gateway failure detection latency for a hard-down provider is roughly 6 s plus the time to fail the in-flight check. For a timeout-based check at 400 ms, add ~1.2 s. So you detect degradation in single-digit seconds.
The cost is real: 240+ models behind one endpoint means many potential health-check connections. If you check each model individually, that is hundreds of TLS handshakes per interval. Most gateways aggregate by provider credential and infer model-level health from provider-level signals, which is usually correct because provider outages are rarely per-model. A cheap GET /v1/models is sufficient to confirm the auth token is valid and the endpoint responds; a tiny completion (e.g., max_tokens=1) confirms the inference path, but costs more.
A gateway such as n4n.ai automatically falls back when a provider is rate-limited or degraded, but the speed of that fallback is governed by the same detection primitives discussed here: passive timeouts plus active probes.
Circuit breakers and half-open states
A bare boolean “up/down” is not enough. You need a circuit breaker with three states:
- Closed: normal traffic.
- Open: fail fast, no calls.
- Half-open: allow a few trial requests to see if the provider recovered.
class Breaker:
def __init__(self, threshold=5, open_ms=5000, half_open_max=3):
self.failures = 0
self.threshold = threshold
self.open_until = 0
self.half_open_trials = 0
def before_call(self):
now = time.monotonic()
if now < self.open_until:
if self.half_open_trials < half_open_max:
self.half_open_trials += 1
return True # permit trial
return False
return True
def on_success(self):
self.failures = 0
self.half_open_trials = 0
self.open_until = 0
def on_failure(self):
self.failures += 1
if self.failures >= self.threshold:
self.open_until = time.monotonic() + 5
self.half_open_trials = 0The half-open state prevents a flapping provider from being permanently banned. It also bounds gateway failure detection latency on recovery: you learn the provider is back within one trial request, not a full health-check cycle. Without half-open, a provider that recovers in 3 s might stay blocked for the full open window (e.g., 30 s), which is its own outage.
Quantifying gateway failure detection latency
Let’s define two metrics:
- T_detect_hard: time from provider hard-down to circuit open.
- T_detect_degraded: time from start of 5xx storm to circuit open.
For passive-only:
- T_detect_hard = request timeout (e.g., 800 ms) + retry routing (<<1 ms).
- T_detect_degraded = (failure_threshold × typical_request_interval) if traffic is sparse; could be minutes under low load.
For active-check + passive:
- T_detect_hard = health_interval × failure_threshold + check_timeout.
- T_detect_degraded = same, because active check samples the error rate.
A hybrid cuts both: passive opens the circuit on the first hard timeout, active checks confirm recovery. In that design, T_detect_hard drops to your request timeout (still bounded by user traffic), but T_detect_degraded drops to the active interval because the checker finds it even when user traffic is low.
If you honor client routing directives (e.g., a header forcing a specific provider), passive detection still applies per route, but active checks must run for all permitted providers or you risk blind spots.
Implementation sketch
A production gateway runs three loops:
- Request path – synchronous, checks breaker state, calls provider with tight timeout, records success/failure.
- Health loop – asynchronous, every
interval_mshits a cheap endpoint, records external failures. - Metrics loop – exports per-provider error rate and latency to Prometheus; alerts on breach.
# Measure raw provider timeout behavior
curl -o /dev/null -s -w "total=%{time_total} code=%{http_code}\n" \
--max-time 0.5 https://provider-api/v1/chat/completions \
-H "authorization: bearer $KEY" \
-d '{"model":"gpt-4o-mini","messages":[]}'If that curl returns code=000 (curl’s failed-connection code) in 2 ms, your passive path will trip instantly. If it hangs for 500 ms then fails, your gateway failure detection latency is capped at 500 ms for that request.
For streaming, add an inter-token watchdog:
async def stream_with_watchdog(provider, req, stall_ms=3000):
last_token = time.monotonic()
async for chunk in provider.stream(req):
last_token = time.monotonic()
yield chunk
if time.monotonic() - last_token > stall_ms / 1000:
raise StreamStall("no token in {}ms".format(stall_ms))Tradeoffs: aggressive vs lazy detection
Aggressive active checking (interval 500 ms, threshold 1) gives sub-second detection but multiplies control-plane traffic. For a gateway fronting dozens of providers, that is manageable. For one fronting 240+ models, you must check at provider granularity or burn CPU and sockets.
Lazy checking (interval 10 s, threshold 5) reduces overhead but lets a degraded provider eat user requests for up to 50 s. In LLM chat use cases, a 50 s outage before reroute is a visible outage.
Passive-only has near-zero overhead but couples detection latency to user timeout. If you set a 10 s timeout to tolerate slow providers, you also tolerate 10 s of dead traffic.
The decisive lever is the request timeout on the critical path. Lower it to 800–1500 ms for interactive endpoints, and let active checks handle the rest. Use a circuit breaker with a low failure threshold (3–5) and a short open window (5 s) so recovery is fast. Forward provider cache-control hints unchanged; they don’t affect detection but preserve cache hits on fallback.
Takeaway
Gateway failure detection latency is not a property of the gateway’s code path; it is a policy you set. Hard failures are caught in milliseconds by passive detection the moment a request times out. Degraded providers are caught in seconds by active health checks. The correct architecture uses both: tight request timeouts with passive circuit breaking for instant hard-down protection, plus out-of-band provider-level health probes at 1–2 s intervals for degradation detection. Set your breaker threshold low, your open window short, and your health checks at provider granularity. Do that, and your users will see a fallback, not an outage.