A 500 from your database is unambiguous: the query failed, the connection dropped, the disk filled. Diagnosing LLM outages vs API outages is fundamentally harder because the failure surface includes non-deterministic model behavior, multi-provider routing, and token-level degradation that traditional APM tools do not capture. If you treat an LLM endpoint like a REST API, you will misclassify partial outages as “slow responses” and waste hours chasing ghosts.
The binary vs the probabilistic
Classical API outages are boolean. Either the service returns 200 with a contract-valid payload, or it does not. Latency may degrade, but the success criterion is stable.
LLM endpoints violate this. A request can return HTTP 200 with a truncated completion, a hallucinated refusal, or a response that silently drops the system prompt due to a provider-side prompt caching miss. The status code is healthy while the output is useless.
# What looks like success
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Summarize the SLA"}],
timeout=10
)
if resp.choices[0].message.content:
# naive check passes, but content may be "I cannot comply"
process(resp.choices[0].message.content)You need semantic validation to catch the failure, which is itself a model call or a regex heuristic that breaks next week.
Provider cascades and hidden dependencies
Most production LLM stacks are not one API. They are a router in front of three providers, each with their own upstreams (model weights, GPU clusters, tokenizer services, safety filters). An outage in one provider’s tokenization service can cause latency spikes that trigger your fallback logic to another provider, which then gets flooded.
Diagnosing LLM outages vs API outages means reconstructing this cascade. A single user-facing error might be the third hop in a retry chain.
{
"trace_id": "a1b2",
"hops": [
{"provider": "A", "status": "timeout", "latency_ms": 10000},
{"provider": "B", "status": "200", "finish_reason": "length", "tokens": 4096}
]
}Without propagated trace context across provider boundaries, you see only the final hop.
Telemetry gaps: tokens, not just status codes
Standard APM records status code, latency, and error rate. For LLMs, those three miss the outage modes that matter:
- Token starvation: provider returns 200 but
finish_reason: "length"because max_tokens was silently capped. - Cache miss storms: system prompt cache key changes due to whitespace, blowing up latency and cost.
- Rate limit by token, not request: you are under request quota but hit a per-minute token ceiling.
What a minimal request looks like
An OpenAI-compatible call returns usage. If you discard it, you are blind.
resp = client.chat.completions.create(
model="mistral-large",
messages=msgs,
temperature=0
)
print(resp.usage)
# {"prompt_tokens": 1200, "completion_tokens": 300, "total_tokens": 1500}A sudden drop in completion_tokens across requests with similar prompts signals a provider truncating outputs. That is an outage no status code will show.
Non-determinism breaks causal inference
With a deterministic API, you reproduce the bug: same input, same output, same failure. LLMs give you none of that.
Two identical requests can yield different finish_reason values due to load-shedding on the provider side. You cannot pin the root cause by replaying production traffic because the provider’s internal state changed.
Repro is not a thing
Suppose your eval suite passed yesterday. Today, the same prompt returns empty content 5% of the time. You suspect your code. You spin up a local proxy to record. By the time you capture, the provider recovered. Postmortem stalls.
This is why diagnosing LLM outages vs API outages demands continuous token-level baselining, not post-hoc repro.
Tradeoffs of fallback and abstraction
Abstraction layers promise reliability. A gateway that automatically routes around a degraded provider feels like a win. It is—until it hides the degradation.
When automatic fallback hides the outage
If your stack silently flips from Provider A to Provider B on A’s 429, your dashboards show 100% success. Meanwhile A is down for 40 minutes. You learn from the provider’s status page, not your own telemetry.
A gateway like n4n.ai can automatically fall back when a provider is rate-limited, but that masks the primary outage from your dashboards unless you export per-token metering and tag the routed provider in your spans. The fallback is a palliative, not a diagnosis.
You must emit the originally attempted provider, the fallback reason, and the token cost delta. Otherwise you trade a visible outage for a mysterious cost spike.
Building diagnostic muscle
You cannot eliminate LLM outages. You can make them legible.
Trace context propagation
Propagate traceparent across every provider call, including retries. If your gateway does not support it, wrap the client.
import openai
from opentelemetry import trace
tracer = trace.get_tracer("llm-client")
with tracer.start_as_current_span("chat") as span:
client = openai.OpenAI(base_url="https://api.example.com/v1")
resp = client.chat.completions.create(model="foo", messages=msgs)
span.set_attribute("llm.tokens.total", resp.usage.total_tokens)Token-level metrics
Record distributions of prompt_tokens, completion_tokens, and finish_reason per model per provider. Alert on:
finish_reason != "stop"rate > 1%- median
completion_tokensdrop > 30% week-over-week prompt_tokensspike without code change (cache miss)
Decisive takeaway
Treat LLM endpoints as probabilistic systems with opaque internals, not as APIs with smarter payloads. Diagnosing LLM outages vs API outages requires token-level telemetry, cross-provider trace propagation, and a willingness to instrument the abstraction layer instead of trusting its green lights. Build the dashboards before the incident, or you will spend the outage guessing.