How server-sent events work for LLM token streaming

Server-sent events (SSE) are a W3C-standardized protocol for server-to-client push over a single HTTP connection. For large language model APIs, understanding how SSE works for LLM token streaming is critical: it is the mechanism that turns a completed generation into a live feed of tokens without polling or bidirectional sockets.

What SSE is at the wire level

SSE is not a separate transport. It is HTTP with a specific content type and a line-based text format. The server sends Content-Type: text/event-stream and keeps the connection open, writing chunks as they become available.

The payload is Unicode text. Each event is a set of fields prefixed by field: value. The only field LLM APIs use consistently is data:. An event ends with a blank line (two newlines). A comment line starting with : can be used as a keepalive.

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive

data: {"id":"chatcmpl-1","choices":[{"delta":{"role":"assistant"}}]}

data: {"id":"chatcmpl-1","choices":[{"delta":{"content":"Hello"}}]}

data: {"id":"chatcmpl-1","choices":[{"delta":{"content":" world"}}]}

data: [DONE]

That is the entire specification. No binary framing, no length prefixes, no multiplexing.

Why LLM providers chose SSE

The alternative was WebSocket, gRPC, or raw chunked JSON. SSE won because it is GET/POST friendly, traverses proxies natively, and requires zero client libraries in browsers (via EventSource). For backend-to-backend calls, it is just an HTTP response you read line by line.

How SSE works for LLM token streaming in practice

When you call an OpenAI-compatible chat completions endpoint with "stream": true, the server does not wait for the full completion. It flushes a JSON object per token (or per few tokens) as data: lines. The client parses each line, extracts choices[0].delta.content, and appends to the UI or buffer.

A typical first event carries the role and an empty content delta. Subsequent events carry incremental text. The stream terminates with data: [DONE] or simply by closing the connection.

{
  "id": "chatcmpl-abc",
  "object": "chat.completion.chunk",
  "created": 1700000000,
  "model": "gpt-4o-mini",
  "choices": [
    {
      "index": 0,
      "delta": {"content": "The"},
      "finish_reason": null
    }
  ]
}

The client must handle finish_reason in the final chunk, which often arrives with empty delta but includes usage metadata if the provider supports it.

Why SSE matters for latency and UX

Token streaming is not cosmetic. Time-to-first-token (TTFT) determines perceived intelligence. A 20-second monolithic response feels broken; the same answer streamed over SSE with 300ms TTFT feels responsive.

SSE also simplifies cancellation. The client closes the TCP connection, and a well-built server aborts the model inference, saving compute. This is harder with polling.

A minimal client in Python

Most HTTP clients can consume SSE; you just iterate lines. Using httpx:

import httpx, json

with httpx.stream(
    "POST",
    "https://api.example.com/v1/chat/completions",
    json={"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Hi"}], "stream": True},
    headers={"Authorization": "Bearer KEY"},
    timeout=None,
) as r:
    for line in r.iter_lines():
        if not line or not line.startswith("data:"):
            continue
        payload = line[len("data:"):].strip()
        if payload == "[DONE]":
            break
        chunk = json.loads(payload)
        delta = chunk["choices"][0]["delta"].get("content", "")
        if delta:
            print(delta, end="", flush=True)

Note timeout=None. Default timeouts kill long streams. Also iter_lines handles the newline delimiting; do not use iter_text and split manually unless you handle partial lines across chunks.

A browser/TypeScript client

EventSource only supports GET, so LLM streaming from the browser normally uses fetch with a readable stream reader.

const res = await fetch("https://api.example.com/v1/chat/completions", {
  method: "POST",
  headers: { "Content-Type": "application/json", Authorization: "Bearer KEY" },
  body: JSON.stringify({ model: "gpt-4o-mini", messages: [{ role: "user", content: "Hi" }], stream: true }),
});

const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split("\n\n");
  buffer = lines.pop() ?? "";
  for (const block of lines) {
    const dataLine = block.split("\n").find((l) => l.startsWith("data:"));
    if (!dataLine) continue;
    const payload = dataLine.slice(5).trim();
    if (payload === "[DONE]") return;
    const json = JSON.parse(payload);
    process.stdout.write(json.choices[0].delta.content ?? "");
  }
}

The double newline split is mandatory. SSE events are separated by a blank line, not just a newline.

Server-side implementation pitfalls

If you deploy your own LLM proxy or gateway, SSE exposes every buffering layer. Nginx, Gunicorn, and cloud load balancers love to buffer responses. Set X-Accel-Buffering: no for Nginx. In Python, flush sys.stdout or use an async StreamingResponse that awaits each token.

Headers must be exact:

Content-Type: text/event-stream
Cache-Control: no-cache, no-transform
Connection: keep-alive
X-Accel-Buffering: no

Missing no-transform can let intermediaries gzip the stream and destroy interactivity.

Reconnection and id

SSE has a built-in retry: field and id: field for resuming. LLM token streams almost never use them because regeneration is cheap and resuming mid-sentence is semantically odd. But if you implement a custom SSE feed (e.g., for status updates), assign IDs and handle the browser’s automatic reconnect, or you will duplicate events.

Common misconceptions about SSE and LLM streaming

“SSE is just chunked JSON.” No. Chunked transfer encoding is a transport detail; SSE is an application protocol with event boundaries and field parsing. You can send chunked JSON without SSE, but then the client must frame it itself.

“EventSource works with POST.” It does not. If your API requires a body (like chat completions), you must use fetch + ReadableStream or a polyfill that opens a GET with query params.

“Streaming removes rate limits.” Streaming changes billing shape, not quota. Tokens are metered as they leave the server. A gateway that provides per-token usage metering will still count every delta against your limit.

“Auto-reconnect saves interrupted generations.” Native SSE reconnect re-requests the resource. For LLM calls, that means a new inference, not a resume. You must persist your own cursor if you need continuity.

“Fallback is transparent.” When a provider is degraded, an inference gateway such as n4n.ai can automatically route to a healthy model, but the stream may change model field mid-response or emit a different finish_reason. Clients that assert strict schema on the first chunk break. Parse defensively.

Debugging SSE locally

curl is the fastest debugger. Use -N to disable buffering:

curl -N https://api.example.com/v1/chat/completions \
  -H "Authorization: Bearer KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hi"}],"stream":true}'

If you see the full response only after completion, a proxy is buffering. Check X-Accel-Buffering and your app server’s flush behavior.

When not to use SSE

If you need bidirectional control (e.g., the server must accept mid-generation corrections from the client), WebSocket or HTTP/2 gRPC are better. For pure token delivery, SSE is the lowest-friction option and is what every major LLM vendor ships.

Understanding how SSE works for LLM token streaming lets you build clients that fail gracefully, proxy layers that do not stall, and UIs that feel instantaneous. The protocol is 15 years old and boring—which is exactly why it works.