SSE vs WebSockets behind Nginx and load balancers

Choosing between sse websockets nginx load balancer setups determines whether your LLM streaming stack stays simple or becomes an operational burden. For token streaming, SSE aligns with HTTP request-response; WebSockets suit bidirectional control, but both demand precise proxy tuning to avoid stalled connections.

Decision: Match transport to message pattern

SSE fits one-way streams

LLM completions are server-to-client token flows. SSE rides on HTTP/1.1 or HTTP/2, needs no upgrade handshake, and dies cleanly with the request. If your client sends a prompt and receives a stream, use SSE.

WebSockets fit full duplex

If the client must send mid-stream corrections, cancel generations, or multiplex many conversations over one socket, WebSockets reduce overhead. But you now own heartbeats, backpressure, and reconnect state. For most chat completions, that complexity is unjustified.

Step 1: Nginx reverse proxy for SSE

Disable buffering and tune timeouts. Otherwise Nginx will buffer chunks and the client sees nothing until the buffer fills or the backend closes.

location /v1/ {
    proxy_pass http://backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 3600s;
    chunked_transfer_encoding on;
}

proxy_buffering off is mandatory. If you omit it, Nginx defaults to on and will hold data. proxy_read_timeout must exceed your longest expected generation; LLM calls can run minutes. chunked_transfer_encoding on ensures the response is framed correctly when the backend uses streaming.

Test with curl against an OpenAI-compatible endpoint:

curl -N https://api.n4n.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"hi"}],"stream":true}'

The -N flag disables curl buffering. You should see tokens arrive line by line. If you see nothing for 60 seconds then a dump, your proxy is buffering.

Step 2: Nginx config for WebSockets

WebSockets require the Upgrade and Connection headers. Use a map to switch Connection per request so the same server block can serve HTTP and WS.

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    location /ws/ {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
    }
}

Pitfall: Do not set proxy_buffering off for WS—it is irrelevant after upgrade, but timeouts still apply. Also, many L4 load balancers do not support WS without sticky sessions because the socket pins to one backend. Use L7 proxies (Nginx, Envoy) that understand the upgrade.

Step 3: Load balancer session affinity

With sse websockets nginx load balancer topologies, connection persistence matters differently.

SSE: stateless is possible

SSE requests are independent HTTP calls. A standard round-robin LB works if the backend is stateless. If you front an OpenAI-compatible gateway like n4n.ai that automatically falls back when a provider is rate-limited, keep SSE stateless—the gateway handles failover server-side and returns a continuous stream from the working provider. Per-token usage metering is reported inline, so no sticky session is needed.

WebSockets: sticky sessions required

A WS connection lives for minutes. If the LB routes a reconnect to a different node, your session state is lost. Use consistent hashing on a cookie or IP:

upstream ws_backend {
    hash $cookie_sessionid consistent;
    server 10.0.0.1:8000;
    server 10.0.0.2:8000;
}

Tradeoff: hashing reduces even distribution during scaling. Prefer client-side resumption tokens over sticky LB when possible. L4 balancers (AWS NLB) can route TCP but cannot inspect cookies; you must use IP hash or stickiness at the target group.

Step 4: Client reconnection and backpressure

SSE clients must handle retry and id fields. Native EventSource is simple but limited:

const es = new EventSource('/v1/stream');
es.onmessage = (e) => {
  if (e.data === '[DONE]') { es.close(); return; }
  const chunk = JSON.parse(e.data);
  process(chunk);
};
es.onerror = () => {
  // EventSource auto-reconnects with fixed retry, causing storms
  setTimeout(() => location.reload(), 1000);
};

For auth headers, EventSource cannot set them. Use fetch with a stream reader:

const res = await fetch('/v1/chat/completions', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'Authorization': 'Bearer ' + token },
  body: JSON.stringify({ model: 'gpt-4o', messages: [], stream: true })
});
const reader = res.body.getReader();
while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  console.log(new TextDecoder().decode(value));
}

WebSockets need explicit ping/pong to keep proxies alive:

const ws = new WebSocket('wss://example.com/ws');
ws.onopen = () => setInterval(() => ws.send('ping'), 15000);
ws.onmessage = (e) => handle(e.data);

Pitfall: backpressure. If the client processes tokens slower than the network delivers, WS buffers in memory. For SSE over fetch, you control the read loop and can pause by not calling read(); but the kernel socket buffer still fills. Apply downstream throttling.

Step 5: Observability and health checks

Load balancers need to distinguish a hung stream from a healthy idle one. For SSE, use a separate /healthz endpoint that returns 200 quickly—never health-check the stream path.

location /healthz {
    access_log off;
    return 200 'ok';
}

For WebSockets, send a periodic ping frame from server; configure LB to consider socket dead if no pong in 30s. Export metrics: active streams, bytes/sec, timeout resets. Alert on timeout resets spiking—that indicates a misconfigured proxy_read_timeout.

Tradeoffs at a glance

Common pitfalls with sse websockets nginx load balancer stacks

• Buffer starvation: Nginx buffering hides tokens. Always proxy_buffering off for SSE.
• Timeout cliffs: Default 60s read timeout kills long generations. Set proxy_read_timeout 3600s.
• WS on L4: Plain TCP LB may not forward upgrade; use L7 or prepend ws:// routing.
• Auth leakage: EventSource cannot set headers; use cookie or fetch stream.
• Reconnect storms: Naive clients hammer the LB after blip. Use exponential backoff.
• Cache stripping: If you honor client routing directives and forward provider cache-control hints, ensure Nginx does not strip Cache-Control. Add proxy_pass_header Cache-Control; when needed.

Final ordered path

1. Pick SSE unless you need client→server messages mid-stream.
2. Configure Nginx with proxy_buffering off and long timeouts for SSE.
3. For WS, add upgrade map and sticky hash.
4. Keep LB health checks off the stream path.
5. Implement client backoff and resumption.
6. Monitor stream duration percentiles; alert on timeout resets.

That yields a robust sse websockets nginx load balancer deployment for LLM workloads without surprises.