WebSockets for voice agents, SSE for chat streaming

Pick the wrong transport and you will burn weeks on reconnection logic that the protocol already solved. For most LLM products, the right call is simple: use websocket voice agent sse chat streaming as a split architecture—WebSockets for interactive voice, Server-Sent Events for token streaming in chat.

Why the transport split exists

A chat completion is a single request that produces a sequence of tokens. The client sends a prompt, the server pushes text until done. There is no need for the client to speak back mid-stream. Server-Sent Events (SSE) map perfectly to that shape: one HTTP request, a long-lived response, and a standardized text/event-stream format.

A voice agent is different. The browser captures microphone audio continuously and must ship those frames to the server while simultaneously receiving transcriptions, LLM tokens, or synthesized speech. You need full-duplex communication from the first byte. WebSockets give you a single bidirectional pipe without re-establishing HTTP requests. Trying to force voice over SSE means polling or auxiliary POST endpoints, which adds latency and complexity you do not want in a real-time conversation.

SSE for chat streaming: the boring correct choice

Start with fetch and a ReadableStream reader. EventSource only supports GET, and most LLM endpoints require a POST with a JSON body. The code below is the entire client side of a streaming chat call:

async function streamChat(messages: {role: string; content: string}[]) {
  const res = await fetch('https://api.example.com/v1/chat/completions', {
    method: 'POST',
    headers: {'Content-Type': 'application/json', 'Authorization': `Bearer ${KEY}`},
    body: JSON.stringify({model: 'gpt-4o-mini', messages, stream: true})
  });
  const reader = res.body!.getReader();
  const decoder = new TextDecoder();
  while (true) {
    const {done, value} = await reader.read();
    if (done) break;
    const chunk = decoder.decode(value, {stream: true});
    for (const line of chunk.split('\n')) {
      if (line.startsWith('data: ')) {
        const data = line.slice(6);
        if (data === '[DONE]') return;
        const json = JSON.parse(data);
        process.stdout.write(json.choices[0].delta.content ?? '');
      }
    }
  }
}

A minimal Python server using FastAPI looks like this:

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import asyncio, json

app = FastAPI()

async def token_gen():
    for tok in ["Hello", " world", "!"]:
        yield f"data: {json.dumps({'choices':[{'delta':{'content':tok}}]})}\n\n"
        await asyncio.sleep(0.1)
    yield "data: [DONE]\n\n"

@app.post("/v1/chat/completions")
async def chat():
    return StreamingResponse(token_gen(), media_type="text/event-stream")

If you route through an OpenAI-compatible gateway such as n4n.ai, the SSE response already carries per-token usage metering and provider cache-control hints, so your billing loop stays simple and you avoid re-implementing cache TTL logic per provider.

SSE pitfalls

• Proxy buffering: NGINX and many CDNs buffer responses. Set X-Accel-Buffering: no and disable compression for the stream path.
• Client abort: Your server generator must catch asyncio.CancelledError (or the equivalent) and stop calling the model, or you pay for tokens nobody reads.
• No client→server after start: If the user hits “stop”, you need a separate POST to cancel, or you close the TCP connection and handle partial state.

WebSockets for voice agents: bidirectional from day one

Voice cannot wait for a new HTTP request to send the next audio chunk. Open a WebSocket, stream PCM or Opus frames as binary messages, and let the server push back partial transcripts or audio. Here is a browser client that captures mic input and ships 16-bit PCM:

const ws = new WebSocket('wss://voice.example.com/agent');
ws.binaryType = 'arraybuffer';

navigator.mediaDevices.getUserMedia({audio:true}).then(stream => {
  const ctx = new AudioContext();
  const src = ctx.createMediaStreamSource(stream);
  const proc = ctx.createScriptProcessor(4096, 1, 1);
  proc.onaudioprocess = (e) => {
    const pcm = e.inputBuffer.getChannelData(0);
    const buf = new Int16Array(pcm.length);
    for (let i=0; i<pcm.length; i++) buf[i] = pcm[i]*32767;
    ws.send(buf.buffer);
  };
  src.connect(proc); proc.connect(ctx.destination);
});

ws.onmessage = (ev) => {
  if (typeof ev.data === 'string') {
    const msg = JSON.parse(ev.data);
    if (msg.type === 'partial') updateInterim(msg.text);
  } else {
    playBinaryAudio(ev.data); // server-sent TTS audio
  }
};

Server side with the websockets library:

import asyncio, websockets, json

async def handler(ws):
    async for message in ws:
        if isinstance(message, bytes):
            # raw pcm from client
            await ws.send(json.dumps({'type':'partial','text':'(asr stub)'}))
        else:
            msg = json.loads(message)
            if msg['type'] == 'start':
                await ws.send(json.dumps({'type':'ready'}))
async def main():
    async with websockets.serve(handler, '0.0.0.0', 8765):
        await asyncio.Future()
asyncio.run(main())

Voice-specific pitfalls

• Backpressure: If you ws.send audio faster than the link carries it, the browser queues buffers and latency climbs. Monitor socket buffered amount.
• Jitter: Network audio arrives unevenly. You need a playout buffer of ~100–200 ms or speech sounds robotic.
• VAD: Server-side voice activity detection must decide when to hit the LLM. Sending every frame to the model wastes tokens; sending too late adds lag.

Tradeoffs you can’t ignore

SSE rides on standard HTTP/2 or HTTP/1.1. It scales behind ordinary load balancers, supports HTTP caching semantics, and needs no special timeout tuning. Its weakness is strictly one-way push.

WebSockets are stateful. A load balancer must either use sticky sessions or a shared pub/sub layer so a reconnect lands on the same conversation context. You also own the ping/pong heartbeat; cloud load balancers drop idle TCP connections after 60–300 seconds.

For a chat product, SSE is lower operational risk. For a voice agent, WebSockets are not optional—they are the only sane way to maintain sub-second round trips while sending and receiving simultaneously.

Ordered implementation path

1. Ship chat with SSE first. Use the fetch+ReadableStream pattern above. Validate token accounting before building anything real-time.
2. Use one OpenAI-compatible endpoint for chat completions. A gateway with automatic fallback when a provider is rate-limited or degraded reduces the SSE chat error surface; you write one client and get redundancy.
3. Stand up a separate WebSocket service for voice. Do not piggyback audio on your chat API. Different scaling, different auth termination, different health checks.
4. Reuse auth tokens but terminate differently. Validate the same JWT at the WS handshake, but map the connection to an in-memory session store, not a stateless request handler.
5. Instrument both paths. Track SSE disconnect rate, WS reconnect frequency, and token throughput per session. Alert when p99 voice latency exceeds your product threshold.

Production pitfalls that bite later

• SSE through a misconfigured CDN will appear to work in dev and stall in prod. Test with curl --no-buffer against the edge.
• WS behind a Layer 4 LB without sticky causes ghost sessions: the client reconnects, the old server still holds state, and you duplicate LLM calls.
• Ignoring client disconnect in generators is the silent money leak. Always cancel upstream model calls on done.
• Sending 1-second audio blobs over WS triggers jitter spikes. Send 20–40 ms frames and let the server assemble.

The split is not dogma—it is the path of least resistance to a system that stays up. Build the websocket voice agent sse chat streaming boundary explicitly, and each side stays simple enough to debug at 3 a.m.