SSE buffering pitfalls behind nginx and proxies

Streaming LLM tokens over HTTP feels simple until an sse buffering nginx proxy sits between your client and backend. The proxy quietly accumulates response chunks and flushes them in batches, converting a live token feed into a stalled pause followed by a wall of text. This guide gives an ordered path to defeat that buffering and keep Server-Sent Events flowing end to end.

1. Know what the proxy actually does to your SSE

SSE is a long-lived text/event-stream response. nginx defaults to proxy_buffering on. It reads the upstream response into memory (or a temp file) up to proxy_buffer_size and only writes to the client when a buffer fills or the upstream closes. For a token stream that means your data: {...} frames pile up silently.

The buffer chain

The data path is: backend socket → nginx proxy buffer → (optional gzip filter) → client socket. Each stage can coalesce writes. The gzip filter has its own 4–8 KB buffer; even with proxy_buffering off, if gzip on and gzip_proxied permits compression, nginx may wait for a compressible block before emitting.

Tradeoff: buffering cuts syscall overhead for static assets. For SSE it destroys the core latency guarantee.

2. Disable response buffering in nginx

Scope a dedicated location for streaming endpoints. Minimal working config:

location /stream/ {
    proxy_pass http://app:8000;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_buffering off;
    proxy_cache off;
    proxy_store off;
    proxy_read_timeout 3600s;
    chunked_transfer_encoding on;
    gzip off;
    add_header X-Accel-Buffering no;
}

X-Accel-Buffering: no is an internal nginx directive. If you cannot edit the location block, the upstream can send this header itself to disable buffering. Set both for defense.

Scoping the config

Do not set proxy_buffering off in location / globally; it hurts throughput for JSON routes. Use a map if you must toggle per route:

map $request_uri $no_buffer {
    ~^/stream/ 1;
    default     0;
}
server {
    location / {
        proxy_buffering $no_buffer;
    }
}

Pitfall: a nested location block does not inherit add_header from a parent if it defines its own. Repeat add_header X-Accel-Buffering no in every streaming block.

3. Stop request buffering for large prompts

SSE is a response stream, but the POST body (the prompt) can be megabytes. nginx buffers client request bodies to temp files before forwarding when proxy_request_buffering on (default). That delays the first token.

location /v1/chat/ {
    proxy_pass http://app:8000;
    proxy_request_buffering off;
    proxy_buffering off;
    client_max_body_size 32m;
    # ...
}

Tradeoff: disabling request buffering pushes backpressure to the backend socket. Your app must read the body concurrently while streaming out the response, or you’ll see upstream timeouts.

4. Set and forward the correct headers

The backend must emit:

Content-Type: text/event-stream
Cache-Control: no-cache, no-transform
Connection: keep-alive
X-Accel-Buffering: no

In a FastAPI app:

from fastapi.responses import StreamingResponse

async def gen():
    yield "data: hi\n\n"

return StreamingResponse(
    gen(),
    media_type="text/event-stream",
    headers={"X-Accel-Buffering": "no", "Cache-Control": "no-cache, no-transform"},
)

When terminating TLS at nginx, forward the headers untouched:

proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass_header X-Accel-Buffering;
proxy_pass_header Cache-Control;

Never add Cache-Control: max-age. A CDN behind nginx will then cache the stream and replay it.

5. Force flushes with SSE comments

Some intermediaries wait for a 1 KB threshold. Send a comment frame every few seconds; clients ignore lines starting with : but the bytes still cross the buffer boundary.

import time

def sse_stream():
    yield ": connected\n\n"
    last = time.monotonic()
    for tok in generate_tokens():
        yield f"data: {tok}\n\n"
        if time.monotonic() - last > 15:
            yield ": ping\n\n"
            last = time.monotonic()

This also keeps idle proxies from closing the connection on a silent 60s timeout.

6. Verify with curl and a raw client

Test the backend directly, then through the sse buffering nginx proxy with -N (no curl buffer):

curl -N -H "Accept: text/event-stream" \
  https://proxy.example.com/stream/ | cat -v

You should see tokens arrive one per line. If they appear in clumps every few seconds, buffering persists. Measure TTFB:

curl -N -o /dev/null -w "ttfb=%{time_starttransfer}\n" \
  https://proxy.example.com/stream/

Python confirmation:

import requests
r = requests.get("https://proxy.example.com/stream/", stream=True)
for line in r.iter_lines():
    if line:
        print(line.decode())  # should print incrementally

If iter_lines() blocks for seconds, inspect proxy_buffering and gzip in the active nginx config (nginx -T).

7. Handle chained proxies and CDNs

Cloudflare, AWS ALB, and Lambda@Edge may re-buffer. Set Cache-Control: no-transform and X-Accel-Buffering: no at the origin. For Cloudflare, disable “Rocket Loader” and avoid HTTP/2 multiplexing coalescing by using a separate subdomain for streams. ALB respects Connection: keep-alive but has its own idle timeout; set proxy_read_timeout 3600s and ALB idle timeout ≥ 3600s.

Tradeoff: disabling CDN features for streaming reduces caching wins on other routes. Separate the stream host from the API host.

8. Gateway and routing directives

When you front an OpenAI-compatible endpoint such as n4n.ai with nginx, the same rules apply. The gateway honors client routing headers and forwards provider cache-control hints; if your proxy strips X-Accel-Buffering or rewrites Cache-Control, streams stall. Pass headers transparently:

location /v1/ {
    proxy_pass https://api.n4n.ai;
    proxy_buffering off;
    proxy_set_header Authorization $http_authorization;
    proxy_pass_header X-Accel-Buffering;
    proxy_pass_header Cache-Control;
}

Per-token metering still works because the byte stream is unchanged; only latency improves.

9. Common pitfalls checklist

• Nested location blocks do not inherit proxy_buffering unless explicitly set; repeat it.
• add_header in a parent context is dropped if a child defines its own add_header. Repeat X-Accel-Buffering.
• gzip on with gzip_proxied any will buffer SSE despite proxy_buffering off. Set gzip off in the stream location.
• proxy_cache must be off; proxy_cache_valid alone does nothing, but proxy_cache_path with inactive can still store if accidentally enabled.
• Logging $upstream_response_time stays high because the connection is long-lived; use buffer= on access_log to avoid disk I/O spikes.
• Client-side fetch with ReadableStream needs getReader(); calling response.json() will hang forever on a stream.

10. Production monitoring

Track TTFB per route with a synthetic stream that emits one event per second. Alert if the gap between events exceeds 3s. In Node:

const res = await fetch('/stream/', { headers: { Accept: 'text/event-stream' } });
const reader = res.body!.getReader();
const dec = new TextDecoder();
while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  process.stdout.write(dec.decode(value));
}

Keep stream routes out of rate-limited locations; SSE connections count as long-lived and will trip limit_conn if scoped too tightly.

That is the ordered path: understand the buffer chain, disable response and request buffering in scoped nginx blocks, emit correct headers and periodic comments, verify with curl -N, and treat every proxy hop as a potential buffer. Get those right and your sse buffering nginx proxy problem disappears.