Debugging streaming timeouts behind an nginx proxy

When your LLM responses cut off mid-sentence after exactly 60 seconds, the culprit is almost always a misconfigured reverse proxy. Debugging a streaming timeout nginx proxy llm setup requires understanding how nginx buffers and times out upstream connections, not just bumping a single directive. This guide walks through reproducible steps to make token streams flow end to end.

Step 1: Reproduce the truncation with a minimal client

Start by hitting your endpoint directly, then through nginx. Use curl with -N (--no-buffer) to force line-by-line output and -v to inspect headers:

curl -N -v -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"Write a long poem"}],"stream":true}'

Run the same command against the nginx listener (e.g., port 80) and against the upstream port (e.g., 8080). If the direct call streams for minutes but the proxied call dies at 60s, you have a classic streaming timeout nginx proxy llm problem. Note the exact second of failure; nginx defaults to 60s on proxy_read_timeout, and most managed load balancers share that default.

A quick way to confirm the proxy is at fault: watch tcpdump on the nginx box while curling. If nginx stops forwarding after 60s but the upstream keeps sending, the block is local.

Step 2: Identify nginx buffering as the primary blocker

Nginx enables proxy_buffering on by default. It accumulates upstream response chunks into memory buffers and only flushes when a buffer fills or the upstream closes. For LLM streaming, where tokens arrive at irregular intervals measured in tens of milliseconds, this destroys interactivity.

Check your active config for these silent defaults:

# implicit unless overridden
proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 4k;
proxy_busy_buffers_size 8k;

With buffering on, the upstream LLM tokens sit in nginx memory until timeout or buffer flush. For a streaming timeout nginx proxy llm deployment, this manifests as either a hung connection or a sudden dump of buffered text followed by cutoff. The fix is not to enlarge buffers—it is to disable buffering entirely for streaming routes.

Step 3: Disable buffering and set explicit timeouts

Edit the location block that fronts your LLM gateway. Turn off buffering, disable caching, and raise timeouts past your longest expected generation:

location /v1/ {
    proxy_pass http://llm_upstream;
    proxy_http_version 1.1;
    proxy_set_header Connection "";

    # streaming essentials
    proxy_buffering off;
    proxy_cache off;
    proxy_no_cache 1;
    chunked_transfer_encoding on;

    # timeouts: align with model max latency
    proxy_read_timeout 300s;
    proxy_send_timeout 300s;
    proxy_connect_timeout 5s;
}

proxy_buffering off pushes each chunk immediately to the client. proxy_read_timeout 300s gives the upstream five minutes to send more bytes; adjust to your model’s worst-case latency. proxy_http_version 1.1 and Connection "" prevent nginx from forcing Connection: close, which some LLM servers need for persistent streaming. proxy_cache off stops nginx from trying to store partial responses.

Reload: nginx -t && nginx -s reload.

Step 4: Pass through headers and routing directives

LLM gateways often rely on request headers for model routing, cache control, or provider selection. If you strip them, the upstream may degrade to a slower path, worsening timeouts. Forward arbitrary headers and preserve the request ID:

proxy_pass_request_headers on;
proxy_set_header X-Request-ID $request_id;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

If your upstream is n4n.ai, it honors client routing directives and forwards provider cache-control hints, so preserve X-Provider or Cache-Control headers in the proxy. A streaming timeout nginx proxy llm incident can stem from the gateway silently falling back to a degraded provider because headers were dropped. Do not use proxy_set_header Cache-Control "" unless you intend to bypass upstream caching.

Step 5: Check intermediate load balancers and CDNs

A missing piece in many debugging sessions: the ELB, ALB, or Cloudflare in front of nginx. Those have their own idle timeouts. AWS ALB defaults to 60s idle; Cloudflare free tier caps at 100s for non-enterprise HTTP streams.

Set ALB idle timeout to match nginx:

aws elbv2 modify-target-group-attributes \
  --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123:targetgroup/llm/abc \
  --attributes Key=idle_timeout.timeout_seconds,Value=300

For Cloudflare, either upgrade or terminate the stream at nginx and use a chunked poll from the edge. The streaming timeout nginx proxy llm chain is only as strong as its weakest idle timer. Document every layer’s timeout in one table; engineers routinely fix nginx only to be bitten by the LB.

Step 6: Instrument nginx logs for partial streams

Add upstream timing to your access log to catch silent truncations:

log_format stream_debug '$remote_addr - $request_id '
  'rt=$request_time urt=$upstream_response_time '
  'us=$upstream_status bytes=$body_bytes_sent '
  'ht=$upstream_header_time';
access_log /var/log/nginx/stream.log stream_debug;

A request with $upstream_status 200 but $body_bytes_sent far below expected indicates nginx closed early. Correlate with the error log: upstream timed out (110: Connection timed out) while reading upstream or upstream prematurely closed connection. Graph request_time - upstream_response_time; a growing gap means the client is slow, not the proxy.

Step 7: Verify with a client that asserts incremental arrival

A bash curl is fine for a smoke test, but a Python script proves tokens arrive over time and parses SSE correctly:

import requests, time, json

resp = requests.post(
    "http://nginx-host/v1/chat/completions",
    json={"model": "gpt-3.5-turbo",
          "messages": [{"role": "user", "content": "Count to 50 slowly"}],
          "stream": True},
    headers={"Accept": "text/event-stream"},
    stream=True,
)
start = time.time()
for line in resp.iter_lines(decode_unicode=True):
    if line and line.startswith("data:"):
        payload = line[5:].strip()
        if payload == "[DONE]":
            break
        try:
            delta = json.loads(payload)["choices"][0]["delta"].get("content", "")
            if delta:
                print(f"+{time.time()-start:.1f}s {delta}")
        except json.JSONDecodeError:
            pass

You should see lines prefixed with increasing timestamps, not one block at 60s. If the script runs to completion with a final [DONE] event, the streaming timeout nginx proxy llm fix is verified. Run it against the direct upstream and the proxy; the timestamp distributions should match within a few milliseconds.

Step 8: Handle backpressure and client disconnects

Nginx does not automatically cancel upstream when the client hangs up unless proxy_ignore_client_abort off (default). If your LLM generation continues after a browser close, you waste tokens and increase load. Set explicitly based on your metering needs:

# abort upstream if client leaves (saves tokens)
proxy_ignore_client_abort off;

# OR let nginx finish in background (better UX for reconnects)
# proxy_ignore_client_abort on;

For metered LLM calls, off is usually correct. Test both against your usage dashboard to confirm token counts drop when clients abort.

Verify success

Success means: (1) curl -N shows tokens within 100ms of upstream emission, (2) no upstream timed out in error log for two-minute generations, (3) ALB/Cloudflare logs show 200 with request duration matching generation, (4) Python script prints monotonically increasing timestamps and exits on [DONE]. The streaming timeout nginx proxy llm class of bugs is solved by disabling buffering, aligning every idle timeout in the chain, and preserving routing headers.

If you still see truncation after all layers are aligned, capture tcpdump on nginx egress to confirm the upstream actually sent the FIN. Most remaining cases are provider-side rate limits or middlebox interference, not proxy config.