Server-sent events explained for LLM APIs

Server-sent events (SSE) is a unidirectional HTTP-based protocol that lets a server push text updates to a client over a single long-lived connection. In the context of an LLM API, SSE is the standard transport for streaming token-by-token responses from models like GPT-4, Claude, or Llama without the overhead of WebSockets or the latency of polling. If you’ve used the OpenAI stream: true parameter, you’ve already consumed an SSE stream.

How SSE works under the hood

At the protocol level, SSE is remarkably simple. The client makes a standard HTTP request with Accept: text/event-stream. The server holds the connection open and writes lines of text formatted as data: <payload>\n\n. Each double newline terminates an event. That’s it — no framing layer, no binary opcodes, no handshake upgrade.

GET /v1/chat/completions HTTP/1.1
Host: api.example.com
Accept: text/event-stream
Authorization: Bearer sk-...

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive

data: {"id":"cmpl-123","object":"chat.completion.chunk","choices":[{"delta":{"content":"Hello"},"index":0}]}

data: {"id":"cmpl-123","object":"chat.completion.chunk","choices":[{"delta":{"content":" world"},"index":0}]}

data: [DONE]

The browser’s native EventSource API handles reconnection automatically. On the server side, you just write to the response stream and flush. No special infrastructure required — any HTTP/1.1 or HTTP/2 server can speak SSE.

The event format

Each event consists of optional fields followed by a blank line:

For LLM streaming, providers typically use only data: with JSON payloads. The [DONE] sentinel (or a final chunk with finish_reason) signals completion.

Why SSE matters for LLM APIs

Token streaming is the default UX

Users expect to see output appear incrementally. A 2,000-token response at 50 tokens/second takes 40 seconds to generate. Blocking on the full response feels broken. SSE delivers tokens as they’re produced with minimal latency — typically one RTT plus model inference time per chunk.

Simpler than WebSockets

WebSockets require a protocol upgrade (Upgrade: websocket), custom framing, ping/pong heartbeats, and a separate code path from your REST API. SSE runs on vanilla HTTP/1.1 or HTTP/2. It works through every proxy, load balancer, and CDN without configuration. You can debug it with curl. You can inspect it in browser dev tools. Your existing middleware (auth, rate limiting, logging) works unchanged.

Native browser support

EventSource has been supported everywhere since IE polyfills. No extra dependencies. The API is tiny:

const evtSource = new EventSource('/api/stream', {
  headers: { Authorization: `Bearer ${token}` }
});

evtSource.onmessage = (e) => {
  const chunk = JSON.parse(e.data);
  if (chunk.choices[0].delta.content) {
    appendToUI(chunk.choices[0].delta.content);
  }
};

evtSource.onerror = () => {
  // EventSource auto-reconnects; handle fatal errors here
};

HTTP/2 multiplexing solves the connection limit

The historical knock on SSE was the browser limit of 6 connections per hostname (HTTP/1.1). With HTTP/2, that limit disappears — multiple SSE streams share a single TCP connection. If your infrastructure terminates TLS at a modern load balancer (ALB, Cloudflare, nginx), you likely already have HTTP/2 to the client.

Concrete example: consuming an OpenAI-compatible stream

Most LLM gateways expose an OpenAI-compatible /chat/completions endpoint with stream: true. Here’s a production-grade consumer in Python that handles reconnection, backpressure, and parsing:

import json
import httpx
from typing import AsyncIterator

async def stream_chat(
    client: httpx.AsyncClient,
    messages: list[dict],
    model: str,
    *,
    max_retries: int = 3,
    timeout: float = 30.0,
) -> AsyncIterator[str]:
    """
    Yield text deltas from a streaming chat completion.
    Handles reconnection on transient failures.
    """
    payload = {
        "model": model,
        "messages": messages,
        "stream": True,
        "stream_options": {"include_usage": True},
    }
    
    attempt = 0
    last_event_id = None
    
    while attempt <= max_retries:
        headers = {"Accept": "text/event-stream"}
        if last_event_id:
            headers["Last-Event-ID"] = last_event_id
        
        try:
            async with client.stream(
                "POST",
                "/v1/chat/completions",
                json=payload,
                headers=headers,
                timeout=timeout,
            ) as response:
                response.raise_for_status()
                
                async for line in response.aiter_lines():
                    if not line.startswith("data:"):
                        continue
                    
                    data = line[5:].strip()
                    if data == "[DONE]":
                        return
                    
                    try:
                        chunk = json.loads(data)
                    except json.JSONDecodeError:
                        continue
                    
                    # Track last-event-id if provider sends it
                    if "id" in chunk:
                        last_event_id = chunk["id"]
                    
                    delta = chunk.get("choices", [{}])[0].get("delta", {})
                    if content := delta.get("content"):
                        yield content
                        
        except (httpx.ReadTimeout, httpx.RemoteProtocolError) as e:
            attempt += 1
            if attempt > max_retries:
                raise
            # Exponential backoff with jitter
            await asyncio.sleep(min(2 ** attempt + random.random(), 30))
            continue
        
        break  # Success — exit retry loop

Key details in this pattern:

• client.stream() — processes the response incrementally without buffering the entire body
• Last-Event-ID header — enables server-side resumption if the provider supports it (OpenAI doesn’t yet, but some gateways do)
• stream_options.include_usage — requests the final usage chunk for token accounting
• Exponential backoff — respects the provider’s retry-after hints if present

Common misconceptions

“SSE doesn’t work over HTTP/2”

False. SSE works fine over HTTP/2. Each stream is a separate HTTP/2 stream within the same connection. The browser’s EventSource uses HTTP/2 automatically when available. The only caveat: some older load balancers buffer SSE responses, defeating streaming. Disable response buffering for /stream paths:

# nginx
proxy_buffering off;
proxy_cache off;

# Apache
ProxyPass /stream/ http://backend/stream/ flush=on

“You need WebSockets for bidirectional communication”

True, but irrelevant for LLM streaming. The client sends one request; the server streams tokens back. That’s unidirectional. If you need client-to-server streaming (e.g., voice input), use a separate WebSocket or HTTP/2 stream — don’t force the response path to carry it.

“SSE can’t handle binary data”

SSE is text-only. If you need binary (audio chunks, image tiles), encode as base64 in the data: field or use a separate WebSocket. For token streaming, JSON over SSE is the right tool.

“EventSource handles everything automatically”

EventSource reconnects on network errors, but it has blind spots:

• No custom headers on reconnect — you can’t send Authorization on the retry request (browser limitation). Workaround: put the token in a cookie or query param, or implement a manual fetch+retry loop like the Python example above.
• No access to HTTP status codes — if the server returns 401 or 429 after the initial 200, EventSource just sees a closed connection and retries. You need a separate health check or a wrapper that surfaces errors.
• No backpressure signal — the server pushes at its own pace. If the client falls behind, the browser buffers indefinitely. For high-volume streams, consider a manual ReadableStream consumer with pipeThrough and a transform that applies backpressure.

“All providers implement SSE the same way”

They don’t. Differences you’ll encounter:

Write your parser defensively. Accept multiple sentinel formats. Treat any non-JSON data: line as a comment/heartbeat. Log raw lines for debugging.

Production considerations

Connection lifecycle

LLM streams can run 60+ seconds. Infrastructure timeouts will kill them:

Set timeouts at every layer. Test with a slow model (or sleep in a mock) to verify.

Token accounting

Streaming makes usage tracking harder — you don’t get the final usage object until the end. Options:

1. Wait for the final chunk — simple, but loses data if the client disconnects early
2. Estimate client-side — count tokens with a local tokenizer (tiktoken, Hugging Face tokenizers) as you render; reconcile with server usage on completion
3. Log server-side — the gateway emits usage on completion regardless of client disconnect; correlate by request ID

Option 3 is most reliable. If you run a gateway, emit usage to your analytics pipeline on finish_reason != null, not on client ack.

Caching and cache-control

SSE responses must not be cached. Always send:

Cache-Control: no-cache, no-store, must-revalidate
Connection: keep-alive
Content-Type: text/event-stream; charset=utf-8

Some CDNs strip Connection: keep-alive on HTTP/2 (where it’s implicit). That’s fine. What matters is Cache-Control: no-store.

Observability

Instrument three metrics per stream:

1. Time to first token (TTFT) — from request start to first data: line
2. Tokens per second — sustained throughput after first token
3. Completion rate — streams that emit a finish reason vs. those that error or timeout

Correlate with model, provider, and request size. TTFT spikes usually mean cold starts or queueing. Throughput drops suggest provider throttling.

When not to use SSE

• Bidirectional realtime — voice chat, collaborative editing, gaming → WebSockets or WebRTC
• High-frequency server pushes (>100 msg/s) — consider WebSocket binary frames or gRPC streaming
• Clients behind restrictive proxies that buffer or drop long-lived HTTP — rare in 2024, but exists in some enterprise environments
• Non-HTTP transports — IoT over MQTT, CoAP, etc.

For the vast majority of LLM chat and completion use cases, SSE is the correct default. It’s simple, standards-based, and supported by every model provider.

TL;DR

• SSE is HTTP-based server push — text/event-stream, one long response, data: lines, double newline delimited
• Use it for LLM token streaming because it’s simpler than WebSockets, works everywhere, and integrates with existing HTTP middleware
• Consume with EventSource (browser) or httpx.stream / fetch + ReadableStream (server)
• Handle reconnection, timeouts, and provider-specific quirks (sentinels, heartbeats, error formats)
• Disable proxy buffering, set generous idle timeouts, and emit usage metrics server-side
• Don’t overthink it — if you’re streaming tokens from an LLM, SSE is the right tool