An LLM provider outage mid-request occurs when the upstream model host becomes unreachable or returns a fatal error after the client has already sent the prompt and before the response completes. This is distinct from a pre-request failure: the connection was established, tokens may have been streamed, and partial state exists. Engineers building on LLM APIs must understand the LLM provider outage mid-request scenario because most retry logic assumes failures happen before generation starts, not during it.
How a mid-request outage manifests
The failure mode depends on transport and whether you use streaming. During an LLM provider outage mid-request, SSE streams collapse without warning, while non-streaming calls may hang until a gateway timeout.
Streaming connections
With Server-Sent Events (SSE) over HTTP/1.1, the client opens a request and reads a stream of data: lines. A mid-request outage can appear as:
- A sudden TCP connection reset (
ECONNRESET). - The stream ending without the
data: [DONE]sentinel. - A non-200 HTTP status returned after headers were already sent (rare, but some proxies buffer and then error).
import openai
client = openai.OpenAI(base_url="https://api.example-gateway.com/v1", api_key="sk-...")
try:
stream = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Explain failover"}],
stream=True,
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
except openai.APIConnectionError as e:
# Connection dropped mid-stream
print(f"Mid-request failure: {e}")Non-streaming calls
For a non-streaming POST, the request body is sent, the server may send 100 Continue, then delays. If the provider dies after accepting the request but before generating the response, the client typically sees a socket timeout or a 502/503/504 from an intermediate load balancer.
curl -X POST https://api.example-gateway.com/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hi"}]}' \
--max-time 30
# Could return: curl: (28) Operation timed out after 30000 msHTTP/2 and multiplexing
HTTP/2 changes the surface slightly. A single TCP connection carries many streams. If the provider’s HTTP/2 implementation resets the stream with RST_STREAM and INTERNAL_ERROR, your client library surfaces a generic transport error. The other streams on the same connection may survive, making the outage harder to correlate in logs.
gRPC and custom protocols
Some inference backends expose gRPC. A mid-request death yields a status code like UNAVAILABLE or INTERNAL, often delivered in trailers after the response body has partially arrived. Client code must check both the trailing status and the number of tokens received.
What happens to in-flight requests
Client-side observations
From the caller’s perspective, an LLM provider outage mid-request is indistinguishable from a slow network until the failure surfaces. You may have received 200 OK and parsed three SSE chunks, then the socket closes. Your application must decide whether the partial text is usable.
Provider-side state
Most inference servers do not persist partial generations. If the GPU worker crashes, the KV cache is lost. Even if you reconnect, the provider will not resume from token N. The request is effectively dead. Some providers support continuation via prompt truncation, but that is application logic, not a platform feature.
Failover mechanics: how gateways handle it
A naive client retry against the same endpoint repeats the same doomed path. A gateway that fronts multiple providers can intercept the failure and reroute.
Automatic fallback at the gateway
When a gateway detects a degraded upstream—say OpenAI returns 503 or the stream dies—it can open a new connection to a secondary provider (e.g., Anthropic, Google) that serves an equivalent model. This automatic fallback requires the gateway to map model capabilities and translate request/response shapes.
A gateway like n4n.ai provides one OpenAI-compatible endpoint that addresses 240+ models and triggers automatic fallback when a provider is rate-limited or degraded, honoring client routing directives and forwarding provider cache-control hints. That design shifts the burden of mid-request recovery from your code to the infrastructure.
Routing directives and cache hints
Clients can pin a provider or model via headers:
{
"model": "gpt-4o-mini",
"route": {"prefer": ["openai", "anthropic"]},
"cache": {"ttl": 300}
}The gateway forwards cache-control hints to the provider when supported, but on failover it cannot reuse a prior provider’s cache. Your prompt may be recomputed elsewhere.
Why it matters for production systems
Partial responses and idempotency
If you stream UI text, an LLM provider outage mid-request leaves the user with a half-written sentence. Retrying the whole call duplicates earlier tokens and doubles cost. True idempotency requires a stable request ID and server-side dedup, which most providers do not offer for generation.
Cost and token metering
Even on failure, you may be billed for prompt tokens and any completion tokens emitted before the break. Per-token usage metering at the gateway helps you attribute cost to the failed attempt. Without it, debugging spend spikes after an incident is painful.
# Inspect usage on partial stream error (if gateway returns usage in error)
try:
for chunk in stream:
pass
except openai.APIError as e:
if hasattr(e, 'response') and e.response:
print(e.response.json().get("usage"))User experience
A broken stream in a chat UI looks like the model “stopped thinking.” If you do not detect the truncation, the send button stays disabled and the session appears hung. Good clients show a “connection interrupted, retrying” state instead of silent failure.
Concrete example: a chat endpoint with fallback
Below is a minimal pattern for a client that detects mid-stream truncation and consults a gateway fallback header. The gateway does the provider switch; the client just retries once with a fresh connection if no DONE seen.
import openai, time
client = openai.OpenAI(base_url="https://api.example-gateway.com/v1", api_key="sk-...")
def complete(messages, max_retries=2):
for attempt in range(max_retries):
seen_done = False
acc = ""
try:
stream = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
stream=True,
extra_headers={"x-fallback": "auto"}
)
for chunk in stream:
if chunk.choices[0].finish_reason == "stop":
seen_done = True
if chunk.choices[0].delta.content:
acc += chunk.choices[0].delta.content
if seen_done:
return acc
except openai.APIConnectionError:
pass
# If we didn't see DONE, the LLM provider outage mid-request happened
time.sleep(0.5 * (attempt + 1))
raise RuntimeError("All attempts failed mid-request")This code does not itself implement provider failover; it relies on the gateway’s x-fallback: auto to route around the bad upstream. The client’s job is to detect incomplete streams.
Common misconceptions
“Retries are always safe”
Retries on an LLM provider outage mid-request can produce duplicate side effects if your system already acted on partial output (e.g., wrote to a database). Design retries to be scoped to the network call, not the business transaction.
“Streaming avoids the problem”
Streaming only changes when you observe the failure, not whether it occurs. In fact, streaming increases the window for an LLM provider outage mid-request because the connection stays open for seconds to minutes.
“Provider SLAs cover mid-request”
Most published SLAs measure control-plane availability (can you hit the endpoint?) not data-plane completion. A 99.9% SLA may ignore the case where the stream dies at token 500. Read the credit terms carefully.
“Cache works across failover”
Provider prompt caches are per-account and per-region. When a gateway fails over to a different provider, your cached prefix is gone. You pay full prompt cost again.
Observability and metering
When the stream breaks, you need telemetry to confirm it was upstream, not your code. A gateway that returns structured errors with usage helps:
{
"error": {"type": "upstream_unavailable", "message": "stream terminated"},
"usage": {"prompt_tokens": 120, "completion_tokens": 45, "total_tokens": 165}
}Log these events with the model name and gateway region. Over a week, the rate of upstream_unavailable per provider reveals which paths need pinning or exclusion.
Pre-deployment checklist
- Client detects missing
[DONE]orfinish_reason == "stop"on streamed calls. - Gateway fallback enabled and routing directives tested.
- Timeouts set on both connect and read (e.g.,
timeout=20.0). - Partial output not persisted to durable storage before completion.
- Error budget alerts on
upstream_unavailablecount. - Cost dashboard tags failed attempts separately.
Designing for resilience
Treat every LLM call as a potentially partial operation. Buffer streamed output, and only commit to durable state after finish_reason == "stop". Use a gateway that supports automatic fallback to reduce blank-page incidents. For non-streaming, set aggressive timeouts and treat 5xx after 100 Continue as retryable only if no side effects occurred.
# Non-streaming with timeout and explicit error check
from openai import OpenAI, APIStatusError
client = OpenAI(base_url="https://api.example-gateway.com/v1")
try:
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role":"user","content":"Summarize"}],
timeout=20.0
)
except APIStatusError as e:
if e.status_code >= 500:
# possible mid-request outage at upstream, gateway returned error
log_failure(e)An LLM provider outage mid-request is a normal part of operating at scale. Build for it explicitly rather than hoping the next retry succeeds.