Handling proxy buffering that breaks LLM streaming

When you put a reverse proxy or load balancer in front of an LLM API, proxy buffering breaks LLM streaming by holding the response body in memory and only flushing large chunks. Instead of tokens arriving one-by-one over SSE, the client gets silent gaps followed by a wall of text. This post walks through a concrete fix you can apply end to end, with runnable commands and a verification script.

Why proxy buffering breaks LLM streaming

LLM endpoints emit Server-Sent Events (SSE) or raw chunked HTTP. Each token is a small frame: data: {"choices":[{"delta":{"content":"hi"}}]}\n\n. A correctly behaving proxy forwards each frame the moment it arrives from upstream. Nginx, however, defaults to proxy_buffering on. It reads the upstream response into buffers and only sends to the client when the buffer fills or the upstream connection closes. The result is exactly what teams complain about: proxy buffering breaks LLM streaming in production even when the model is fast.

Other culprits exist: HAProxy with strict timeouts, Caddy’s default flush interval, Envoy’s per-route buffer limits, and application servers like Gunicorn if misconfigured. The fix is always the same: tell every hop to forward bytes immediately.

Step 1: Reproduce the issue with a raw client

Before changing config, prove the buffering exists. Run a direct request to the upstream (bypass the proxy) and then through the proxy.

# Direct to upstream (assuming localhost:8080)
curl -N -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-3.5-turbo","stream":true,"messages":[{"role":"user","content":"count to 10"}]}'

You should see tokens appear every few tens of milliseconds. Now hit your proxy (e.g., http://localhost):

curl -N -X POST http://localhost/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-3.5-turbo","stream":true,"messages":[{"role":"user","content":"count to 10"}]}'

If the second command prints nothing for 3 seconds and then dumps all lines at once, you have buffering.

For programmatic proof, use Python:

import requests, time

t0 = time.time()
resp = requests.post(
    "http://localhost/v1/chat/completions",
    json={"model":"gpt-3.5-turbo","stream":True,"messages":[{"role":"user","content":"count to 10"}]},
    stream=True
)
for line in resp.iter_lines():
    if line:
        print(f"+{(time.time()-t0)*1000:.0f}ms {line[:30]}")

A healthy stream shows incremental timestamps; a buffered one shows a cluster at the end.

Step 2: Locate the buffering layer

Run curl -v through the proxy and inspect headers. If you see X-Accel-Buffering: no missing and the connection is Keep-Alive, nginx is likely buffering. Check your nginx site config:

grep -R "proxy_buffering" /etc/nginx/

If it returns nothing, the default is on. Also check for proxy_cache and gzip directives—both can force buffering.

If you’re behind a CDN (Cloudflare, Fastly), they may buffer too. Cloudflare does not buffer SSE if you set Cache-Control: no-transform and avoid their minification, but verify with the same curl test. A service mesh like Istio/Envoy can also inject buffering; check the virtual service config.

Step 3: Disable buffering in nginx

Edit the location block that proxies to your LLM service. Add explicit directives:

location /v1/ {
    proxy_pass http://upstream_llm;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_buffering off;
    proxy_cache off;
    proxy_set_header X-Accel-Buffering no;
    # Disable gzip for streaming to avoid compression buffering
    gzip off;
    # Ensure chunked transfer is used
    chunked_transfer_encoding on;
}

The X-Accel-Buffering: no header is critical: it signals nginx internally to disable buffering even if a higher-level config re-enables it. Without this, a parent http block can still buffer. Reload:

nginx -t && systemctl reload nginx

Otherwise proxy buffering breaks LLM streaming behind nginx even with proxy_buffering off in the wrong scope.

Step 4: Fix other common proxies

Caddy (v2) buffers by default with a 1s flush interval. Set flush_interval to -1:

reverse_proxy /v1/* upstream_llm {
    flush_interval -1
    header_up X-Accel-Buffering no
}

HAProxy does not buffer responses, but if you use option http-server-close it will close the server connection after each request; for streaming you want option http-keep-alive and timeout tunnel 1h. Also avoid compression algo gzip on the SSE path.

Envoy requires a zero buffer limit on the route:

route:
  retry_policy: {}
  timeout: 3600s
  buffer_limit: 0

Gunicorn (if used as the app server) should run with --worker-class gevent --worker-connections 1000 and never use the sync worker for streaming. Sync workers can buffer wsgi.output behind the scenes.

Step 5: Verify with a latency histogram

Re-run the Python script from Step 1. Success means inter-chunk delays under ~200ms and a smooth progression. A quick verification snippet:

import requests, time, statistics

resp = requests.post("http://localhost/v1/chat/completions",
    json={"model":"gpt-3.5-turbo","stream":True,"messages":[{"role":"user","content":"write 100 words"}]},
    stream=True)
deltas = []
last = time.time()
for line in resp.iter_lines():
    if line:
        now = time.time()
        deltas.append(now - last)
        last = now
print(f"chunks={len(deltas)} median_delta={statistics.median(deltas)*1000:.1f}ms max={max(deltas)*1000:.1f}ms")

If median_delta is under 100ms and max under 500ms, streaming is fixed. If you still see max in the thousands, something upstream is still buffering—likely a second proxy or the LLM provider’s own gateway.

Step 6: Handle client-side and library pitfalls

Some SDKs accidentally re-buffer. The OpenAI Python client sets stream=True and uses iter_lines, which is fine. But if you wrap it in a response.json() or collect into a list before rendering, you defeat the purpose. In JavaScript:

const res = await fetch("/v1/chat/completions", {
  method: "POST",
  body: JSON.stringify({ model: "gpt-3.5-turbo", stream: true, messages }),
  headers: { "Content-Type": "application/json" }
});
const reader = res.body!.getReader();
const decoder = new TextDecoder();
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  process.stdout.write(decoder.decode(value)); // immediate
}

Do not await res.text() then split—that buffers the whole response.

Compression is another silent killer. If nginx has gzip on globally, it will wait for the buffer to compress. The gzip off; in Step 3 prevents this. For Caddy, avoid encode gzip in the streaming route. Also ensure your load balancer health checks are not piggybacking on the streaming port with HTTP/1.0 requests that force closure.

Step 7: When the gateway already streams

If you route through n4n.ai, its OpenAI-compatible endpoint streams tokens directly from providers and honors cache-control hints; the only place you’ll encounter the problem is your own ingress proxy. The steps above apply regardless of which upstream you use.

For other managed gateways, check their docs for X-Accel-Buffering support. Some set it automatically; others require a query param.

Verification checklist

• curl -N shows tokens immediately through the proxy.
• Python delta script reports median < 100ms.
• No X-Accel-Buffering missing in response headers (curl -v shows X-Accel-Buffering: no).
• Gzip disabled on the streaming location.
• Second proxy (CDN, service mesh) also configured with buffering off.

If all boxes are ticked, proxy buffering breaks LLM streaming no longer. You’ve restored real-time generation to your users.