n4nAI
Open console →
API integrationHow-toError Handling & Status Codes

Streaming errors in LLM APIs: how to detect and recover

Practical guide to llm api streaming error handling: detect SSE failures, parse error events, implement reconnection, and recover from truncated streams.

n4n Team4 min read884 words

Audio narration

Coming soon — every post will get a voice note here.

LLM API streaming error handling is messy because Server-Sent Events hide failures behind a persistent connection. A dropped TCP socket or a provider-side 503 can truncate a response mid-token, leaving your parser half-fed and your users staring at a blank box. You need explicit detection and recovery logic because most SDKs only surface the happy path.

Step 1: Distinguish transport errors from protocol errors

A streaming response fails in two fundamentally different ways. The TCP connection dies (timeout, reset, proxy hang), or the server sends a valid SSE frame that describes an error. Your recovery strategy differs for each, so log them separately from the start.

Transport errors throw at the I/O layer. In Python with requests, you’ll see ConnectionError or ChunkedEncodingError while iterating response.iter_lines(). In Node, the ReadableStream emits an 'error' event. In Go, http.Response.Body returns io.ErrUnexpectedEOF on a truncated chunked body.

Protocol errors arrive as data. OpenAI-compatible APIs send data: {"error": {...}} before closing, or an event: error frame. Ignore those at your peril—they often carry a structured type field (rate_limit, context_length_exceeded) that informs whether a retry is even worth it.

import requests

class TransportError(Exception):
    pass

def stream_completion(url, headers, payload):
    with requests.post(url, headers=headers, json=payload, stream=True, timeout=30) as r:
        if r.status_code != 200:
            raise RuntimeError(f"HTTP {r.status_code}: {r.text}")
        for line in r.iter_lines(decode_unicode=True):
            if not line:
                continue
            if line.startswith("data: "):
                yield line[6:]
            elif line.startswith("event: error"):
                raise RuntimeError(f"SSE error event: {line}")

Good llm api streaming error handling starts by separating these two causes in your telemetry. A spike in TransportError means network or provider capacity; a spike in protocol errors means your payloads or quota.

Step 2: Parse SSE frames and surface error events

SSE is not JSON-over-HTTP. Frames are prefixed with data: and terminated by a blank line. A completion stream ends with data: [DONE]. If you see a JSON object with an error key, the stream is dead. Write a parser that never assumes the next line is valid.

Strip the prefix, handle event: and id: lines, and default to ignoring unknown fields. Don’t use split("\n") naively—proxies sometimes coalesce frames. Buffer until you see two consecutive newlines.

import json

def parse_sse_line(line):
    if line.startswith("data: "):
        payload = line[6:]
        if payload == "[DONE]":
            return ("done", None)
        try:
            obj = json.loads(payload)
        except json.JSONDecodeError:
            raise ValueError(f"Malformed SSE data: {payload!r}")
        if "error" in obj:
            return ("error", obj["error"])
        return ("chunk", obj)
    return ("meta", line)

In TypeScript, the same logic applies to response.body.getReader():

const reader = response.body!.getReader();
const decoder = new TextDecoder();
let buffer = "";
while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  const frames = buffer.split("\n\n");
  buffer = frames.pop() ?? "";
  for (const frame of frames) {
    const line = frame.split("\n").find(l => l.startsWith("data: "));
    if (!line) continue;
    const data = line.slice(6);
    if (data === "[DONE]") return;
    const obj = JSON.parse(data);
    if (obj.error) throw new Error(obj.error.message);
  }
}

If you skip this step, a provider that returns {"error":{"type":"rate_limit"}} will look like a silent truncation. That blind spot is the most common bug in llm api streaming error handling deployments.

Step 3: Implement bounded retry with jitter

When the transport drops, retry. But blind retries hammer a degraded provider and duplicate generations. Use exponential backoff with full jitter and a max attempt count.

import time, random

def stream_with_retry(url, headers, payload, max_attempts=4):
    for attempt in range(max_attempts):
        try:
            return list(stream_completion(url, headers, payload))
        except (requests.ConnectionError, requests.ChunkedEncodingError) as e:
            if attempt == max_attempts - 1:
                raise TransportError("exhausted retries") from e
            sleep = (2 ** attempt) * 0.1 + random.uniform(0, 0.2)
            time.sleep(sleep)

Treat a 429 or 503 status as retryable, but respect the Retry-After header if present. For llm api streaming error handling, a 429 means stop and wait; a 503 mid-stream means the backend died and you should reconnect to a different replica.

If you route through an inference gateway such as n4n.ai, automatic fallback to a healthy provider occurs when a provider is rate-limited or degraded. Your client still must catch the interrupted stream and re-issue the request; the gateway will route it to a different backend, often without changing your code.

Step 4: Recover partial completions without duplicating side effects

LLM streams are typically not resumable. If you lose the connection after 50 tokens, you cannot ask the API to continue from token 51 on the same completion. You have two options: restart the whole request, or switch providers and accept a different continuation.

Restart is safe only if your downstream consumer can handle a fresh prefix. For chat UIs, prepend the already-rendered text to the new stream and hide the duplicate. For tool calls, buffer the full response before executing.

def resilient_stream(url, headers, payload):
    seen = []
    while True:
        try:
            for chunk in stream_completion(url, headers, payload):
                seen.append(chunk)
                yield chunk
            return
        except TransportError:
            payload = {**payload, "prompt_suffix": "".join(seen)}
            continue

Gateways that forward provider cache-control hints (for example, n4n.ai honors client routing directives and forwards provider cache-control hints) can make the restarted request cheaper if the prefix matches cache keys. That turns a painful retry into a near-free replay.

This pattern is a core part of llm api streaming error handling for production chat systems where a blank box is a churned user.

Step 5: Detect truncated streams and heal the JSON

A stream that ends without data: [DONE] is truncated. If you were assembling structured output (JSON mode), you now hold invalid JSON. Don’t discard it—attempt a repair.

Use a tolerant parser that closes open strings and brackets. Python’s json.loads will not do this; use a small fallback or json5 if you control the schema.

def safe_json_loads(partial):
    try:
        return json.loads(partial)
    except json.JSONDecodeError:
        healed = partial.rstrip()
        if healed.count('"') % 2: healed += '"'
        if healed.count("{") > healed.count("}"): healed += "}"
        if healed.count("[") > healed.count("]"): healed += "]"
        return json.loads(healed)

For non-JSON text, just flag the truncation and show a “response interrupted” state. Robust llm api streaming error handling means the user sees a retry button, not a spinner that never ends.

Step 6: Verify with fault injection

You cannot claim your llm api streaming error handling works until you break the network on purpose. Run a local proxy that closes the connection after N bytes.

# using nc to simulate a truncated stream
printf 'HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\n\r\n' | \
  (head -c 200; sleep 1; kill -PIPE $$) | nc -l 8080

Point your client at http://localhost:8080 and confirm:

  1. The client detects the broken pipe.
  2. It retries with backoff.
  3. If you send a full SSE error frame instead, the client raises a typed error.

Add a unit test that feeds your parser a truncated data: line and asserts the healer runs. Only then trust the code in production.

Step 7: Meter and alert on stream failures

Per-token usage metering is useless if 10% of streams die silently. Track two counters: stream_transport_error and stream_protocol_error. Alert when the ratio exceeds 1% over five minutes.

If you use a gateway that honors client routing directives, log the x-request-id and the provider that served the stream. That lets you pinpoint which backend is flaky and route around it before your users notice.

Good streaming reliability is not about avoiding errors—it’s about making them invisible to the user. Implement the steps above, test against real interruptions, and your LLM integration will survive provider hiccups without a blank text box.

Tagsstreamingerror-handlingssereliability

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
API integrationHow to build a circuit breaker for LLM API callsBuild a circuit breaker for LLM API calls in Python to stop cascading failures, with timeout, fallback, and half-open state code examples.API integrationIdempotency keys for LLM API requests: why they matterLearn why idempotency keys llm api prevent duplicate charges and inconsistent outputs on retries, with a practical implementation guide for engineers.API integrationTimeout tuning for LLM API requests: best practicesPractical guide to LLM API request timeout best practices: set connect and read timeouts, use retries and streaming, and avoid common latency pitfalls.API integrationRetry with exponential backoff for LLM API callsImplement a robust exponential backoff llm api retry pattern in Python: handle 429s, 5xx, and network errors with jitter, caps, and tests.API integrationStructured error handling for multi-provider LLM routingPractical patterns for multi-provider LLM error handling: typed exceptions, retry budgets, fallback chains, and parsing provider status codes correctly.API integrationContent moderation errors: handling 400 flagged responsesLearn how to detect and handle content moderation 400 error llm api responses with practical steps, code samples, and verification tips for engineers.

All error handling & status codes posts →