Streaming tokens from a language model over HTTP feels simple until a response runs past 30 seconds and your client silently disconnects. Getting the sse timeout keep-alive llm configuration right is the difference between a robust integration and a flaky one that drops long completions mid-sentence. This guide walks through the exact settings you need at each layer, from the model endpoint to your application code.
1. Know the SSE contract
Server-Sent Events are a simple text protocol over a long-lived HTTP connection. The server sends Content-Type: text/event-stream, Cache-Control: no-cache, and Connection: keep-alive. The client reads chunks until the stream closes, usually signaled by data: [DONE]. For LLM outputs, a token arrives every few milliseconds to seconds; gaps are normal, not errors.
A common mistake is treating the stream like a regular REST call with a fixed 10-second total timeout. If the model thinks for 20 seconds before emitting, your HTTP client aborts the connection. The fix is to separate connect timeout from read timeout, and to disable any idle read timeout or set it far above expected inter-token gaps.
Standard tools betray you here. curl --max-time 30 counts the entire transfer, so a 200-second stream fails. Use curl -N --max-time 300 instead. Python’s requests library has no per-chunk read timeout when streaming; you need httpx or aiohttp.
2. Server and gateway keep-alive settings
If you run your own reverse proxy in front of an OpenAI-compatible endpoint, default proxy read timeouts will kill streams. In nginx, turn off buffering and raise timeouts:
location /v1/chat/completions {
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
chunked_transfer_encoding on;
}The proxy_read_timeout 300s tells nginx to wait five minutes between data chunks before giving up. For an sse timeout keep-alive llm deployment, that single line prevents most silent truncations. proxy_buffering off stops nginx from waiting for a full buffer before forwarding, which would defeat streaming.
Envoy uses idle_timeout:
route:
timeout: 300s
idle_timeout: 300sCaddy is simpler:
reverse_proxy backend {
flush_interval -1
transport http {
read_timeout 300s
}
}Don’t set timeouts to infinity. A hung backend should eventually be recycled. Five minutes covers the vast majority of long RAG or agentic responses.
3. Client-side configuration
Python with httpx
httpx exposes explicit timeouts per phase:
import httpx
timeout = httpx.Timeout(connect=5.0, read=300.0, write=5.0, pool=5.0)
with httpx.Client(timeout=timeout) as client:
with client.stream("POST", "https://api.example.com/v1/chat/completions",
json={"model": "gpt-4o", "messages": [], "stream": True}) as r:
for line in r.iter_lines():
if line.startswith("data:"):
print(line[5:])The read=300.0 is the per-chunk read timeout. As long as the server sends any byte within 300 seconds, the connection stays open.
For async code, use aiohttp:
import aiohttp
async def main():
timeout = aiohttp.ClientTimeout(total=None, sock_connect=5, sock_read=300)
async with aiohttp.ClientSession(timeout=timeout) as sess:
async with sess.post(url, json=payload) as resp:
async for line in resp.content:
if line.startswith(b"data:"):
print(line.decode())TypeScript with fetch
Browser EventSource only supports GET, so LLM APIs use POST via fetch. Set an AbortController with a generous timeout:
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), 300_000);
const res = await fetch("https://api.example.com/v1/chat/completions", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ model: "gpt-4o", messages: [], stream: true }),
signal: controller.signal,
});
const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buffer = "";
while (true) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const events = buffer.split("\n\n");
buffer = events.pop() ?? "";
for (const ev of events) if (ev.startsWith("data:")) console.log(ev.slice(5));
}
clearTimeout(timer);In Node.js, undici (the built-in fetch) respects headersTimeout and bodyTimeout in the agent. Set them high:
import { Agent, fetch } from "undici";
const agent = new Agent({ headersTimeout: 300_000, bodyTimeout: 300_000 });
await fetch(url, { method: "POST", body, dispatcher: agent });4. Proxy and gateway layers
Corporate proxies and cloud load balancers often enforce their own idle cutoffs. AWS ALB default idle timeout is 60 seconds; raise it to 300 or place a TCP keepalive ping. GCP HTTPS LB has a 600s timeout but buffers unless you set timeout-sec and use --stream. Cloudflare’s free tier may buffer; enterprise plans pass through with no-cache.
If you sit behind an inference gateway that aggregates providers, behavior during failover matters. For example, n4n.ai provides automatic fallback when a provider is rate-limited or degraded; during a switch it may pause the token stream briefly. Your sse timeout keep-alive llm client must tolerate a multi-second gap without resetting. The gateway still honors your stream: true and forwards provider cache-control hints, but the TCP connection stability is your responsibility at the client.
5. Keep-alive heartbeats
Some LLM servers send periodic SSE comments (: ping) to keep intermediaries from timing out. If yours doesn’t, and you control the server, inject one every 15 seconds:
@app.route("/stream")
def stream():
def gen():
for i in range(100):
yield f"data: {i}\n\n"
if i % 10 == 0:
yield ": ping\n\n"
time.sleep(2)
return Response(gen(), mimetype="text/event-stream")If you can’t change the server, implement a client-side stall detector. Tradeoff: heartbeats add bandwidth but defeat proxy idle cuts; a watchdog adds code complexity but wastes no bytes.
6. Detecting stalled streams
A robust pattern wraps the reader with a deadline per chunk. In Python:
import asyncio, httpx, time
async def stream_with_watchdog(url, json, stall=30.0, hard=300.0):
async with httpx.AsyncClient(timeout=httpx.Timeout(None)) as client:
async with client.stream("POST", url, json=json) as r:
last = time.monotonic()
async for line in r.aiter_lines():
last = time.monotonic()
# external task checks `last` and alerts after `stall`
# aborts connection only after `hard`
print(line)Set the HTTP read timeout to hard (300s) and alert after stall (30s). This separates “model is thinking” from “connection is dead” without prematurely killing a valid generation.
7. OS and network tweaks
Linux closes idle TCP connections via net.ipv4.tcp_keepalive_time (default 7200s). That’s usually fine, but stateful NAT firewalls can evict mappings after 60–120s of no packets. Enable TCP keepalive on the socket:
import socket
sock = socket.socket()
sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 60)
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 30)On macOS, TCP_KEEPALIVE is the equivalent option. This prevents home-router NAT table eviction for very long streams.
8. Testing your configuration
Don’t wait for a production 10k-token response to discover a 60s cap. Simulate with a local Flask app:
from flask import Response, stream_with_context
import time
@app.route("/stream")
def stream():
def gen():
for i in range(100):
yield f"data: {i}\n\n"
time.sleep(2) # 200s total
return Response(stream_with_context(gen()), mimetype="text/event-stream")Then run a client against it:
curl -N --max-time 300 -X POST http://localhost:5000/streamIf it stops at 30 or 60, the timeout is upstream of your app code. Use tcpdump or nginx logs to confirm where the FIN originates.
Common pitfalls and tradeoffs
- EventSource + POST: Browser
EventSourcecannot send a JSON body. Usefetch+ReadableStreamas shown. - Buffering proxies: Cloudflare free tier, some WAFs, and default nginx buffering will delay or truncate streams.
- Too-long timeouts: Setting
read=86400hides backend hangs. Keep it bounded (300s) and rely on a watchdog for observability. - Keep-alive comments: Sending
: pingevery 15s defeats proxy idle cuts but adds ~0.1% bandwidth for long sessions. - HTTP/2 multiplexing: If your endpoint uses HTTP/2, a single stalled stream can block others on the same connection; use separate connections or HTTP/1.1 for streaming.
Tuning sse timeout keep-alive llm settings is not glamorous, but it is the difference between a demo that works and a product that survives real workloads. Start with proxy timeouts, set client read timeout to 300s, and add a watchdog for observability.