Detecting provider degradation from rising p95 latency

When an upstream model provider starts failing silently, the first signal is usually a shift in tail latency. Detecting provider degradation p95 latency before it burns your error budget requires disciplined measurement and a baseline you trust, not a one-off curl test.

Step 1: Instrument every request with provider and model tags

You cannot manage what you cannot attribute. If your gateway aggregates latency across all backends, a slow provider hides behind fast ones. Tag each completion call with the resolved provider and model. Most OpenAI-compatible gateways return the serving provider in a response header (commonly x-provider or x-upstream). Capture it at the call site.

import time
import logging

def timed_completion(client, **kwargs):
    start = time.perf_counter()
    resp = client.chat.completions.create(**kwargs)
    elapsed = time.perf_counter() - start
    provider = resp.headers.get("x-provider", "unknown")
    model = kwargs.get("model", "unknown")
    logging.info({
        "event": "llm_request",
        "provider": provider,
        "model": model,
        "latency_ms": round(elapsed * 1000, 1),
        "usage_tokens": resp.usage.total_tokens if resp.usage else 0,
    })
    return resp

For streaming responses, split the metric: measure time_to_first_token from the first chunk, and inter_token_latency from the chunk deltas. A provider can have a healthy p95 for full response but stall on the first token, which users feel as “dead air.”

Ship these logs to a system that supports label-based aggregation. Plain stdout works for local dev; for production, forward to Prometheus via a pushgateway or use OpenTelemetry spans with the same labels.

Step 2: Export latency as a histogram, not a gauge

A single p95 computed in app code is fragile and loses resolution when you change window sizes. Use a Prometheus histogram so you can recompute quantiles over arbitrary windows without resampling raw events.

from prometheus_client import Histogram

llm_latency = Histogram(
    "llm_request_duration_seconds",
    "End-to-end LLM request latency",
    ["provider", "model", "stream"],
    buckets=(0.2, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0)
)

Bucket boundaries should match your SLOs. If your p95 normally sits at 800ms, a bucket at 1.0 gives you one bit of resolution around the target; add finer buckets (0.7, 0.8, 0.9) if you need early warning.

The PromQL for p95 per provider over 5 minutes:

histogram_quantile(
  0.95,
  sum(rate(llm_request_duration_seconds_bucket[5m])) by (le, provider)
)

This query is the raw material for detecting provider degradation p95 latency across your fleet. Run it as a recording rule to keep dashboards cheap.

Step 3: Establish a rolling baseline

Absolute thresholds lie. A 2-second p95 is fine for a 70B model behind a cold cache but disastrous for a tiny classifier. Compute a trailing baseline from the previous 24 hours, excluding windows where an incident was already declared.

import numpy as np

def rolling_p95(latencies: list[float], window: int = 5000) -> float:
    if len(latencies) < 2:
        return float("nan")
    sample = latencies[-window:]
    return float(np.percentile(sample, 95))

# Load last 24h of latencies for provider "openai" from your TSDB
baseline = rolling_p95(historical_latencies, window=5000)

Store the baseline per provider/model pair in a low-latency KV store. Refresh it every hour using a job that ignores windows where an alert was active, so a week-long degradation doesn’t become the new “normal.” If you have sparse traffic for a model, widen the window or fall back to a global provider average.

Step 4: Alert on relative deviation, not absolute spikes

Page only when the current p95 exceeds the baseline by a meaningful margin for a sustained period. A transient blip should not wake anyone.

def is_degraded(current_p95: float, baseline_p95: float, mult: float = 1.5, min_abs: float = 1.0) -> bool:
    if np.isnan(current_p95) or np.isnan(baseline_p95):
        return False
    return current_p95 > max(baseline_p95 * mult, baseline_p95 + min_abs)

# In your checker loop:
if is_degraded(current_p95, baseline_p95):
    trigger_alert(provider="openai", current=current_p95, base=baseline_p95)

For Prometheus, encode the same logic with a recording rule that compares short and long windows:

- record: job:llm_p95_ratio
  expr: |
    histogram_quantile(0.95, sum(rate(llm_request_duration_seconds_bucket[5m])) by (le, provider))
    /
    histogram_quantile(0.95, sum(rate(llm_request_duration_seconds_bucket[6h])) by (le, provider))

Alert when job:llm_p95_ratio > 1.5 for 10 minutes. Require two consecutive windows to fire to kill single-deploy hiccups. This approach to detecting provider degradation p95 latency avoids both alert fatigue and missed slow-rolls.

Step 5: Route around the rotten backend

Detection is half the job. The fix is to stop sending traffic to the degraded provider. If you run your own gateway, implement weighted routing that drops providers above the degradation threshold. If you use a managed OpenAI-compatible endpoint, leverage its fallback behavior. n4n.ai, for instance, performs automatic fallback when a provider is rate-limited or degraded, but you should still emit your own signals to confirm the fallback is actually engaging and not silently retrying the bad path.

To test routing directives, send a forced provider header and measure latency separately:

from openai import OpenAI

client = OpenAI(base_url="https://your-gateway.example/v1", api_key="key")

# Force a specific provider for canary testing
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "ping"}],
    extra_headers={"x-provider-preference": "azure"}
)
print("served-by:", resp.headers.get("x-provider"))

If the forced provider shows clean p95 while the default route is degraded, your detection logic is validated and the gateway’s routing layer is trustworthy.

Verify success

You know the pipeline works when:

1. Synthetic degradation triggers an alert. Use a fault-injection proxy to add 3s latency to one provider. Within two window lengths, the p95 ratio crosses 1.5 and the alert fires.
2. Baseline recovers after incident. Once you remove the injection, the rolling baseline catches up within 24h and false positives stop.
3. Traffic shifts. In a gateway with fallback, logs show fewer requests to the slow provider and no increase in 5xx errors.

A quick local check:

# Generate traffic, inject latency on provider B via env
LATENCY_INJECT_MS=3000 PROVIDER=openai python load_test.py --p95-expect-under 2000

If the script exits non-zero while your dashboard shows the alert active for that provider, detecting provider degradation p95 latency is operational in your environment.

Caveats

p95 hides the p99. If your users complain about occasional 30-second hangs while p95 looks fine, track p99 alongside. Provider degradation often pairs with token-streaming stalls—measure time-to-first-token separately from inter-token latency, because a stuck stream with a healthy final p95 still ruins the UX.

Finally, never alert on a single window. A provider can hiccup for 30 seconds during a regional deploy. Require sustained deviation across two or three consecutive 5-minute windows before paging a human. Keep the feedback loop tight: measure, baseline, alert, reroute, verify.