What is streaming in an LLM API?

Streaming in an LLM API is a response mode where the server emits tokens as they are generated, rather than buffering the full completion. The client receives a sequence of partial responses over a persistent HTTP connection, typically formatted as Server-Sent Events (SSE). This transforms the request from a blocking RPC into a progressive data stream.

How streaming works at the protocol level

When you send a chat completion request with "stream": true, the server holds the HTTP connection open and writes chunks as the model produces them. Each chunk is a small JSON object wrapped in an SSE frame:

POST /v1/chat/completions HTTP/1.1
Content-Type: application/json
Authorization: Bearer sk-...

{
  "model": "gpt-4o-mini",
  "messages": [{"role": "user", "content": "Explain streaming"}],
  "stream": true
}

The response headers signal streaming semantics:

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
Transfer-Encoding: chunked

Each event follows the SSE format — a data: line with JSON, terminated by a blank line:

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1700000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"role":"assistant","content":"Streaming"},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1700000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":" delivers"},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1700000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":" tokens"},"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1700000000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

The delta field contains only the new token(s). The final chunk has finish_reason: "stop" (or "length", "tool_calls", etc.) and an empty delta. The literal [DONE] sentinel marks stream termination.

Why streaming matters for production systems

Perceived latency and time-to-first-token

Non-streaming requests block until the entire generation completes. For a 500-token response at 50 tokens/second, that’s 10 seconds of silence. Streaming delivers the first token in tens of milliseconds — the model’s time-to-first-token (TTFT) — and paints the rest progressively. Users perceive the system as responsive even when total generation time is identical.

Backpressure and memory bounds

Streaming lets you process tokens incrementally. You can render to a UI, write to a file, or pipe to another service without buffering the full response in memory. This matters for long generations (code files, reports, multi-turn conversations) where the complete output may exceed comfortable heap limits.

Cancellation and speculative execution

With a persistent connection, the client can abort early — close the socket, send AbortController.abort() — and the server stops generating. This enables patterns like speculative decoding: start generating, show the user, and cancel if they intervene or navigate away.

Token-level observability

Each chunk carries metadata: token IDs, logprobs (if requested), finish reasons. You can measure per-token latency, detect anomalies mid-stream, and attribute costs precisely. Non-streaming responses collapse this visibility into a single aggregate.

Concrete example: a resilient streaming client

Production clients must handle reconnection, partial parsing, and backpressure. Here’s a minimal but complete Python implementation using httpx and asyncio:

import asyncio
import json
import httpx
from dataclasses import dataclass
from typing import AsyncIterator, Optional


@dataclass
class StreamChunk:
    content: str
    finish_reason: Optional[str]
    raw: dict


async def stream_chat(
    client: httpx.AsyncClient,
    url: str,
    headers: dict,
    payload: dict,
    *,
    max_retries: int = 3,
    retry_delay: float = 1.0,
) -> AsyncIterator[StreamChunk]:
    """Yield parsed chunks with automatic retry on transient failures."""
    attempt = 0
    while attempt <= max_retries:
        try:
            async with client.stream("POST", url, headers=headers, json=payload, timeout=None) as resp:
                resp.raise_for_status()
                async for line in resp.aiter_lines():
                    if not line or not line.startswith("data: "):
                        continue
                    data = line[6:].strip()
                    if data == "[DONE]":
                        return
                    chunk = json.loads(data)
                    choice = chunk["choices"][0]
                    delta = choice.get("delta", {})
                    content = delta.get("content", "")
                    finish_reason = choice.get("finish_reason")
                    yield StreamChunk(content, finish_reason, chunk)
                return  # successful completion
        except (httpx.RequestError, httpx.HTTPStatusError) as e:
            attempt += 1
            if attempt > max_retries:
                raise
            await asyncio.sleep(retry_delay * attempt)  # exponential backoff

Key details in this implementation:

• client.stream() returns an AsyncClient context manager that yields the response without reading the body
• aiter_lines() parses SSE frames line-by-line, avoiding full-body buffering
• Retries with exponential backoff handle transient provider blips (rate limits, brief degradations)
• The StreamChunk dataclass gives callers structured access while preserving the raw payload for debugging

A TypeScript equivalent using the standard fetch API:

interface StreamChunk {
  content: string;
  finishReason: string | null;
  raw: unknown;
}

async function* streamChat(
  url: string,
  headers: Record<string, string>,
  payload: object,
  options: { maxRetries: number; retryDelayMs: number } = { maxRetries: 3, retryDelayMs: 1000 }
): AsyncGenerator<StreamChunk> {
  let attempt = 0;
  while (attempt <= options.maxRetries) {
    try {
      const resp = await fetch(url, {
        method: "POST",
        headers: { "Content-Type": "application/json", ...headers },
        body: JSON.stringify(payload),
      });
      if (!resp.ok || !resp.body) {
        throw new Error(`HTTP ${resp.status}`);
      }
      const reader = resp.body.getReader();
      const decoder = new TextDecoder();
      let buffer = "";
      while (true) {
        const { done, value } = await reader.read();
        if (done) break;
        buffer += decoder.decode(value, { stream: true });
        const lines = buffer.split("\n");
        buffer = lines.pop() ?? "";
        for (const line of lines) {
          if (!line.startsWith("data: ")) continue;
          const data = line.slice(6).trim();
          if (data === "[DONE]") return;
          const chunk = JSON.parse(data);
          const choice = chunk.choices[0];
          const content = choice.delta?.content ?? "";
          const finishReason = choice.finish_reason ?? null;
          yield { content, finishReason, raw: chunk };
        }
      }
      return;
    } catch (e) {
      attempt++;
      if (attempt > options.maxRetries) throw e;
      await new Promise(r => setTimeout(r, options.retryDelayMs * attempt));
    }
  }
}

Common misconceptions

“Streaming is just a UI trick”

False. Streaming changes the failure domain. A non-streaming request that fails at token 499 of 500 wastes all prior compute and forces a full retry. A streaming request that fails at token 499 has already delivered 498 tokens to the client — the user sees partial output, the system can resume from a checkpoint, and you avoid recomputing the prefix. This is a fundamental reliability difference, not cosmetic.

“You need WebSockets for true streaming”

SSE over HTTP/1.1 or HTTP/2 works fine for unidirectional server-to-client streams. WebSockets add bidirectional complexity (ping/pong, frame masking, custom protocol) that LLM completions don’t need. SSE reconnects automatically in browsers, works through most proxies, and maps cleanly to fetch/axios/httpx. Reserve WebSockets for cases where the client must send interleaved messages (e.g., collaborative editing with live model suggestions).

“Streaming disables caching”

Provider-side caching (prompt prefix caching, KV cache reuse) operates independently of the response transfer mode. The model either hits a cached prefix or it doesn’t. Streaming only affects how the generated tokens are delivered. Some providers even emit cache-hit metadata in the first chunk — you get the signal before the first token renders.

“All providers implement streaming identically”

The OpenAI-compatible SSE format is a de facto standard, but deviations exist:

• Some providers omit role in subsequent deltas (only present in the first chunk)
• Tool call streaming varies: some emit tool_calls as an array of incremental objects, others as a single growing string
• Finish reasons differ: "stop", "length", "tool_calls", "content_filter", "function_call" (legacy)
• A few providers send heartbeat comments (: keep-alive\n\n) during long gaps

Write your parser defensively. Treat the schema as a contract with optional fields, not a rigid type.

“Streaming increases token costs”

Token pricing is identical. You pay for generated tokens whether they arrive in one payload or fifty. The only cost difference is negligible HTTP overhead (extra headers per chunk). In fact, streaming can reduce effective cost by enabling early cancellation — you stop paying for tokens the user never sees.

When not to stream

Streaming adds client complexity. Avoid it when:

• The completion is short and latency-insensitive (classification, embedding, structured extraction with response_format)
• You need atomic all-or-nothing semantics (e.g., generating a JSON object that must validate before use)
• The client environment cannot handle persistent connections (some serverless functions, edge runtimes with strict duration limits)
• You’re batching thousands of requests where connection overhead dominates

For these cases, non-streaming with a reasonable timeout is simpler and more robust.

Integration notes for gateway layers

If you operate a proxy or gateway in front of multiple providers, streaming requires careful handling:

• Header forwarding: Pass Accept: text/event-stream upstream. Strip Content-Length — chunked transfer encoding is mandatory.
• Timeouts: Configure separate connect, read, and idle timeouts. A 30-second read timeout kills legitimate long generations; use an idle timeout (e.g., 10 seconds between chunks) instead.
• Buffering: Disable response buffering in your proxy (nginx: proxy_buffering off;). Buffered proxies defeat the purpose by holding chunks until the stream ends.
• Fallback: If an upstream provider fails mid-stream, you cannot transparently fail over — the client has already received partial output. Design your routing to select a healthy provider before the stream starts, or accept that mid-stream failures surface as errors to the client.
• Usage metering: Count tokens from the stream deltas in real time. The final usage field may arrive in the last chunk or a separate event; don’t wait for it to enforce quotas.

Streaming is the default for interactive LLM workloads because it aligns the protocol with the model’s sequential generation process. The implementation effort pays off in perceived latency, cancellation semantics, and observability. Build your client once, test it against provider quirks, and treat the stream as the primary contract — not an optimization layered on top.