SSE vs WebSockets for streaming chat completions

When building real-time LLM interfaces, the decision of sse vs websockets chat completions shapes your client architecture, infra complexity, and failure modes. The practical debate is not about which protocol is newer, but which matches the request/response shape of LLM inference and your deployment constraints. Most OpenAI-compatible APIs emit Server-Sent Events; WebSockets offer a bidirectional channel that few LLM providers natively expose.

Capabilities: Unidirectional vs Full-Duplex

SSE is a one-way server-to-client push over an established HTTP response. The client opens a request (usually POST with a JSON body) and the server holds the connection open, emitting data: lines as tokens generate. The client cannot send further data on that same stream; to send the next message it issues a new HTTP request. Cancellation is done by aborting the fetch.

WebSockets start with an HTTP Upgrade handshake, then become a full-duplex frame channel. After the socket is open, both sides can transmit arbitrarily. For chat completions, this means you can send a prompt, receive token frames, and then without renegotiating, push a “stop” command or a follow-up instruction on the same socket.

The capability gap matters for agents that need mid-generation control. If your product lets users interrupt a stream with a button that must immediately alter server behavior, WebSockets give you a clean in-band signal. With SSE you either open a second POST /cancel endpoint or tear down the connection and rely on the model stopping early.

Cost and Infrastructure Model

Transport choice drives backend cost more than token price. SSE rides on standard HTTP semantics. A stateless renderer can run on Cloudflare Workers, AWS Lambda, or a horizontally scaled container fleet behind any load balancer. Each chat turn is an independent request; the server pays for compute only while producing tokens.

WebSockets require the server to maintain a socket object in memory for the life of the session, which may span idle thinking time between user turns. You need sticky sessions or a shared message bus if you run more than one node. On managed platforms, WebSocket support often means a dedicated WebSocket API (e.g., API Gateway WS) with its own billing dimensions and cold-start characteristics.

Token metering is identical regardless of transport—providers count prompt and completion tokens the same. A gateway that records per-token usage does not care whether the bytes arrived via SSE or WS. The delta is purely operational: WS holds resources longer.

Latency and Throughput Characteristics

First-byte latency is dominated by model inference, not transport. Both SSE and WS run over TLS; WS adds one extra round trip for the Upgrade, but that cost is amortized across the session. For a typical chat completion where the client sends a prompt and waits for the first token, SSE over HTTP/2 with connection reuse is within a millisecond of WS.

Throughput is comparable. SSE frames are text lines prefixed with data: and separated by blank lines; parsing is trivial. WebSocket frames have a 2–14 byte header and no textual framing overhead. At token sizes of a few bytes to a few dozen bytes, the difference is noise. If you stream thousands of tiny control events per second between client and server, WS has a slight edge; chat completions rarely do.

Ergonomics: Client and Server Code

Browser Client

SSE for chat requires fetch with a streaming body reader because the native EventSource API only supports GET. The pattern is straightforward:

const res = await fetch('https://api.example.com/v1/chat/completions', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ model: 'gpt-4o', messages, stream: true }),
});
const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buffer = '';
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split('\n\n');
  buffer = lines.pop() ?? '';
  for (const line of lines) {
    if (line.startsWith('data: ')) {
      const payload = line.slice(6);
      if (payload === '[DONE]') return;
      const json = JSON.parse(payload);
      process(json.choices[0].delta.content ?? '');
    }
  }
}

WebSockets use the built-in WebSocket object, which feels simpler for bidirectional flows:

const ws = new WebSocket('wss://api.example.com/v1/chat');
ws.onopen = () => ws.send(JSON.stringify({ messages }));
ws.onmessage = (e) => {
  if (e.data === '[DONE]') return;
  const json = JSON.parse(e.data);
  process(json.choices[0].delta.content ?? '');
};
ws.onerror = (e) => console.error(e);

Server Implementation

A minimal SSE endpoint in FastAPI:

from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

async def token_gen():
    for tok in ["Hello", " world", "!" ]:
        yield f"data: {tok}\n\n"
    yield "data: [DONE]\n\n"

@app.post("/chat")
async def chat():
    return StreamingResponse(token_gen(), media_type="text/event-stream")

The equivalent WebSocket handler using websockets:

import asyncio, json
from websockets import serve

async def handler(ws):
    async for msg in ws:
        data = json.loads(msg)
        # ignore data in this toy example
        for tok in ["Hello", " world", "!" ]:
            await ws.send(json.dumps({"choices":[{"delta":{"content":tok}}]}))
        await ws.send("[DONE]")

async def main():
    async with serve(handler, "localhost", 8765):
        await asyncio.Future()

asyncio.run(main())

SSE servers fit naturally into existing HTTP frameworks. WebSocket servers need an event loop and often a separate port or path.

Ecosystem and Protocol Support

The LLM ecosystem has standardized on SSE. OpenAI’s /chat/completions with stream: true returns text/event-stream. Anthropic, Cohere, and every OpenRouter-class gateway follow suit. This means curl, Postman, and browser fetch all work without special clients.

WebSockets are common in gaming and chat apps, but almost no public LLM inference API exposes a WebSocket native interface. If you route through a gateway such as n4n.ai, the OpenAI-compatible SSE endpoint fronts 240+ models with automatic fallback on provider degradation, so you inherit SSE’s interoperability without writing provider-specific WS adapters.

On the client side, SSE over fetch integrates with React Query, SWR, and standard AbortController cancellation. WebSocket state management usually requires a custom hook to handle reconnect, backpressure, and message ordering.

Hard Limits and Operational Constraints

Browser connection limits are the first ceiling. Under HTTP/1.1, a browser allows roughly six simultaneous connections per host; if you open six streaming chats, a seventh blocks. HTTP/2 raises this dramatically via multiplexing, but many LLM endpoints still terminate at HTTP/1.1 edges. WebSockets also consume a connection slot but stay open longer, increasing pressure.

Proxy and firewall behavior differs. SSE traverses standard HTTPS proxies, though some intermediaries buffer responses unless you emit heartbeats or disable buffering. WebSockets can be blocked by deep-packet-inspection firewalls that dislike the Upgrade handshake, particularly in strict corporate environments.

Payload constraints: SSE is UTF-8 text only. That is fine for JSON token deltas. WebSockets can send binary, which is useful for audio or image streams but unnecessary for text chat completions.

Comparison Table

Which to Choose: Verdict by Use Case

Standard chat UI with request/response turns – Use SSE. The client sends a prompt, renders tokens, and on the next user message sends a new POST. You get universal provider support, easy cancellation, and zero socket state.

Long-lived agent or voice interface needing mid-stream control – If you own both server and client and must send intermittent commands (temperature change, tool-call approval) without opening new HTTP requests, WebSockets reduce latency and complexity on the wire. Expect to build reconnection and scaling logic.

Serverless or edge deployment – SSE wins outright. It runs on stateless compute and scales to zero. WebSockets on serverless platforms require specialized APIs and incur idle socket charges.

Multi-model routing through a gateway – Stick with SSE. The dominant OpenAI-compatible surface area means you can swap models or providers without touching transport code. As noted, a gateway exposing SSE across hundreds of models with fallback removes the only real argument for WS: provider lock-in.

High-frequency bidirectional token exchange (e.g., collaborative editing with LLM suggestions) – WebSockets if the message rate between peers exceeds what repeated POSTs can comfortably handle. For purely model-to-user token flow, this case is rare.

The sse vs websockets chat completions decision is mostly solved by ecosystem gravity. SSE is the path of least resistance for 95% of LLM products; WebSockets earn their keep only when the client must talk back as fluidly as the server streams.