Tracking streaming completion rates and dropped connections

Streaming LLM responses over HTTP introduces failure modes that classic request/response monitoring misses. If you operate a service that consumes token streams, tracking streaming completion rates dropped connections is the only way to see partial failures, silent truncations, and client aborts that never surface as HTTP 500s.

Define a stream completion event

A stream is complete when the upstream sends a terminating chunk with a finish_reason and the TCP connection closes cleanly. Anything else—connection reset, timeout before final chunk, client cancel—is a drop.

In the OpenAI streaming format, the last relevant chunk looks like:

{
  "choices": [
    {
      "delta": {},
      "finish_reason": "stop",
      "index": 0
    }
  ],
  "id": "chatcmpl-123",
  "object": "chat.completion.chunk"
}

If you only count HTTP 200 responses, you will miss cases where the connection died mid-stream and the client never parsed a finish_reason. That gap is exactly why streaming completion rates dropped connections must be measured at the application layer, not the load balancer.

What counts as a drop

• Upstream closes TCP socket before sending finish_reason.
• Read timeout expires while waiting for the next chunk.
• Client disconnects (browser close, mobile background).
• Proxy or gateway returns 502/503 after streaming starts.

All four reduce the effective completion rate even if the initial HTTP status was 200.

Instrument the client stream reader

Wrap your streaming call so every chunk increments a counter and a timer starts on the first byte. Capture the asyncio.CancelledError or equivalent so intentional client disconnects are logged as drops, not ignored.

import time
import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI()

async def stream_with_metrics(messages, model):
    start = time.monotonic()
    chunks = 0
    finished = False
    try:
        stream = await client.chat.completions.create(
            model=model, messages=messages, stream=True
        )
        async for chunk in stream:
            chunks += 1
            if chunk.choices[0].finish_reason:
                finished = True
        duration = time.monotonic() - start
        if finished:
            # record completion
            pass
        else:
            # record dropped stream (no finish_reason)
            pass
    except asyncio.CancelledError:
        # client disconnected
        finished = False
        raise
    except Exception:
        # upstream/network error
        finished = False
        raise
    finally:
        # emit metric: chunks, duration, finished flag
        pass

Pitfall: SDK exception masking

Many SDKs swallow connection resets into a generic APIError after the fact. You must inspect whether any finish_reason was seen. If not, count it as a drop even if the exception says “stream closed”. Do not rely on the exception type alone.

Idle timeout handling

Set a per-chunk idle timeout (e.g., 30s). If no chunk arrives within that window, force-close and mark the stream dropped. A slow stream that eventually finishes is not a drop; a stalled one is.

Log stream lifecycle at the proxy

If you front requests with a reverse proxy or API gateway, emit a structured log line when the upstream stream ends or the client socket closes. Below is a minimal FastAPI middleware pattern that tracks active streams and logs termination cause.

from fastapi import Request
import logging

logger = logging.getLogger("stream_monitor")

@app.middleware("http")
async def stream_tracker(request: Request, call_next):
    if request.url.path == "/v1/chat/completions":
        response = await call_next(request)
        # real impl uses request.is_disconnected() in a background task
        return response
    return await call_next(request)

A more robust approach uses request.is_disconnected() in a background task that watches the stream coroutine. When the client goes away, you log client_abort. When upstream closes early, you log upstream_drop.

Why proxy logs are incomplete

Standard access logs record the response status after the handler returns. For streaming, the handler may return 200 before the stream body is fully sent. You need either an asynchronous response wrapper or an eBPF socket observer to catch mid-stream resets.

Compute the completion rate metric

Export two counters to Prometheus: streams_started_total and streams_completed_total. The streaming completion rates dropped connections ratio is streams_completed_total / streams_started_total over a window.

# prometheus metrics example
streams_started_total{model="gpt-4o"} 1420
streams_completed_total{model="gpt-4o"} 1390
streams_dropped_total{model="gpt-4o",cause="upstream"} 22
streams_dropped_total{model="gpt-4o",cause="client"} 8

In Grafana, plot:

sum(rate(streams_completed_total[5m])) / sum(rate(streams_started_total[5m]))

Tradeoff: label cardinality

Adding high-cardinality labels like user_id will explode your time-series database. Keep labels to model and cause at most, and push detailed traces to a sampling-based tracer.

Histogram of stream duration

Also record stream_duration_seconds as a histogram. A sudden increase in duration alongside stable drop rate can indicate provider degradation that has not yet crossed the drop threshold.

Separate client aborts from upstream failures

A browser navigating away triggers a TCP FIN from the client. That is not a provider reliability problem, but it still lowers your raw completion rate. Distinguish by logging who closed the socket.

With nginx proxying to an OpenAI-compatible backend, the $upstream_response_time and $request_completion variables help:

log_format stream '$time_local $status $upstream_status '
                  '$upstream_response_time $request_time '
                  '$connection $request_completion';

If $request_completion is “ABORTED” but $upstream_status is 200, the client left early. If $upstream_status is 502 and $upstream_response_time is short, the provider dropped the connection.

Counting these separately keeps your streaming completion rates dropped connections dashboard honest.

Apply fallback and retry carefully

Retrying a dropped stream is not like retrying a POST. If you resend the same prompt, the model generates from scratch, wasting tokens and latency. Better: on upstream drop before finish_reason, reconnect and append the last K tokens you received as context, or use a gateway that supports resume.

An OpenAI-compatible endpoint that addresses 240+ models with automatic fallback when a provider is rate-limited or degraded can mask some drops—but only if it forwards the partial stream state or restarts transparently. Verify that the fallback path does not silently duplicate tokens.

async def resilient_stream(messages, model, max_retries=2):
    last_tokens = []
    for attempt in range(max_retries):
        try:
            async for chunk in stream_with_metrics(messages, model):
                last_tokens.append(chunk)  # simplified
                yield chunk
            return
        except UpstreamDrop:
            messages = messages + [{"role": "assistant", "content": "".join(last_tokens)}]
            continue

Tradeoff: structured output

Resuming via prompt append breaks if the model already emitted a tool call or structured output. For JSON mode, you often must restart entirely and validate the full object. Factor that into retry logic: if finish_reason is missing and you are in structured mode, full restart is safer.

Set SLOs and alert on drops

Define an error budget: e.g., 99% of started streams for a given model complete within 10s of the expected length. Alert when the drop rate exceeds 1% over 5 minutes for production models.

alert: HighStreamDropRate
expr: |
  sum(rate(streams_dropped_total{cause="upstream"}[5m]))
  / sum(rate(streams_started_total[5m])) > 0.01
for: 5m
labels:
  severity: page

Pitfall: absolute counts

Common pitfall: alerting on absolute drop counts during traffic spikes. A 50-drop spike at 2% of 2500 streams is different from 50 drops at 20% of 250 streams. Always use ratios.

Pitfalls and tradeoffs summary

• Chunk timeouts vs drops: A slow stream that eventually finishes is not a drop. Use a per-chunk idle timeout (e.g., 30s) to classify stalls as drops only after the timeout fires.
• SSE vs WebSocket: Server-Sent Events over HTTP/1.1 is the default for OpenAI-compatible APIs. WebSocket gives you bidirectional control to detect client cancel faster, but adds complexity.
• Debug storage cost: Saving full stream payloads for every drop is expensive. Sample 5% of drops and store truncated prefixes.
• SDK buffering: Some client SDKs buffer chunks internally; if you measure at the SDK boundary you may miss network-level resets that occur between buffer flushes.
• Metering mismatch: Per-token usage reported by the provider may count tokens sent before a drop. Reconcile billed tokens against completed streams to catch silent truncation that the provider still charged for.

Tracking streaming completion rates dropped connections is not a one-time task. Wire these metrics into your deploy pipeline so a new model version or provider change shows up as a movement in the completion ratio before users complain.