The SSE streaming differences across LLM APIs are larger than most teams expect when they first wire up a chat loop. What looks like a uniform text/event-stream mask hides incompatible event schemas, divergent tool-call chunking, and inconsistent usage reporting that will surface as bugs the moment you add function calling.
Wire format divergence
Every provider speaks SSE on the wire, but the payload shapes disagree at the field level. The SSE spec allows event:, data:, id:, and retry: lines; how each vendor uses them is where the fracture starts. OpenAI popularized a delta-based JSON object wrapped in data: lines, terminated by data: [DONE], and ignores event: entirely. Anthropic uses named events (message_start, content_block_delta, message_stop) with typed data payloads and relies on the event: field to discriminate. Gemini streams bare JSON data: chunks without a standardized sentinel and often pads with comment lines (: keep-alive) that naive parsers mistake for data.
// OpenAI-compatible chunk
data: {"choices":[{"delta":{"content":"Hello"},"finish_reason":null}]}
// Anthropic chunk
event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}
// Gemini chunk (simplified)
data: {"candidates":[{"content":{"parts":[{"text":"Hello"}]}}]}If you write a single on_message handler assuming choices[0].delta.content, you will silently drop Anthropic text and mis-parse Gemini’s parts. The SSE streaming differences across LLM APIs force either per-provider parsers or a normalization layer before your business logic sees a token.
Tool call streaming is where it breaks
Text streaming is forgiving. Tool calls are not. OpenAI splits a function call across many chunks: the first carries tool_calls[0].id and function.name, subsequent chunks carry incremental function.arguments string fragments. You must concatenate arguments before JSON parsing, and the fragment boundaries do not respect JSON token boundaries.
async def accumulate_tools(stream):
calls = {}
async for chunk in stream:
for tc in chunk.choices[0].delta.tool_calls or []:
idx = tc.index
calls.setdefault(idx, {"id": "", "name": "", "arguments": ""})
if tc.id: calls[idx]["id"] = tc.id
if tc.function.name: calls[idx]["name"] += tc.function.name
if tc.function.arguments: calls[idx]["arguments"] += tc.function.arguments
# arguments may still be partial JSON until finish_reason == "tool_calls"
return callsAnthropic emits a content_block_start with type: "tool_use" containing name and id, then input_json_delta fragments inside content_block_delta. The shape is different but the concatenation problem is identical. Mistral and Groq expose OpenAI-compatible surfaces, yet they may send tool_calls with non-contiguous index values when parallel calls are involved, so your merge map must be keyed by index, not by arrival order.
event: content_block_start
data: {"type":"content_block_start","index":1,"content_block":{"type":"tool_use","id":"toolu_1","name":"get_weather"}}
event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"{\"lat\":"}}The SSE streaming differences across LLM APIs mean your abstraction must model a “streaming tool call” object that accepts fragments from either schema. Miss this and you will ship malformed arguments to your function executor.
Usage metadata and final aggregates
OpenAI attaches usage only on the final non-[DONE] chunk (or via response headers). Anthropic sends message_delta with usage.input_tokens and output_tokens at the end, and separately reports cache_read_input_tokens. Gemini includes usage_metadata inside the last chunk. None of these are guaranteed per-chunk.
If you meter per-token cost client-side, you must wait for stream end and merge. A naive counter that tallies len(content) underestimates token cost and breaks on cached prompt discounts. OpenAI exposes prompt_tokens_details.cached_tokens; Anthropic exposes cache fields natively. When a gateway like n4n.ai fronts multiple providers, it normalizes usage into the OpenAI-shaped usage object and applies per-token metering, but your client still needs to read it from the terminal chunk rather than inferring from deltas.
Error propagation mid-stream
SSE has no built-in per-event error code. OpenAI may abruptly close the stream on a provider 429; Anthropic sends an error event with type and message. Gemini can terminate with a non-200 before stream start, or inject a data: with error field. Heartbeats complicate this: some providers send : ping comments every 15–30s to keep proxies alive, and your reader must ignore them without treating them as EOF.
const res = await fetch(url, { headers });
if (!res.ok) throw new Error(`upstream ${res.status}`);
const reader = res.body!.getReader();
const decoder = new TextDecoder();
let buf = "";
while (true) {
const { done, value } = await reader.read();
if (done) break;
buf += decoder.decode(value, { stream: true });
const lines = buf.split("\n");
buf = lines.pop()!;
for (const line of lines) {
if (line.startsWith(":")) continue; // comment/heartbeat
if (line.startsWith("event: error")) { /* handle named error */ }
if (line.startsWith("data: ")) {
const json = JSON.parse(line.slice(6));
if (json.error) throw new Error(json.error.message);
}
}
}Retries must be idempotent. If you resend a tool-call request after a mid-stream failure, the model may produce different arguments. Design your agent loop to treat interrupted streams as failed turns, not partial state.
Structured outputs and partial JSON
Streaming JSON mode (or tool calls returning JSON) tempts teams to parse incrementally for UI. Don’t. Until finish_reason signals completion, the concatenated string is almost always invalid JSON. Providers differ on whether they emit whitespace or escape characters across chunk boundaries.
Anthropic’s input_json_delta can split a token mid-Unicode-escape. OpenAI’s arguments fragments are arbitrary substrings. You must buffer the full string, then JSON.parse. If you need progressive rendering of structured data, use a streaming JSON parser that tolerates prefixes, but keep it strictly read-only until the terminal event.
Client buffering and timeouts
Streaming clients must manage backpressure and read deadlines. A slow consumer that does not drain res.body will stall the provider connection and trigger upstream timeouts. Use an AbortController with a generous signal timeout (e.g., 60–120s for long completions) and always exhaust the reader in a finally block. SSE libraries that buffer entire responses in memory before yielding are unacceptable for multi-minute agent runs; prefer async iterators that yield per line.
async def stream_lines(response):
async for raw in response.content.aiter_lines():
if raw.startswith(":"):
continue
yield rawThe gateway normalization tradeoff
Building adapters for each provider is maintenance tax. A gateway such as n4n.ai collapses these variances behind one OpenAI-compatible endpoint that addresses 240+ models, automatically falling back when a provider is rate-limited or degraded. That removes most client-side branching, but it does not eliminate the need to understand the underlying quirks: cache-control hints, tool-call fragmentation, and usage accounting still follow provider semantics underneath.
If you skip the gateway, you own the parsers, the fallback logic, and the version drift when Anthropic adds a new event type. If you use one, you trade transparency for velocity and must still write defensive stream consumption.
Decisive takeaway
Treat SSE streaming differences across LLM APIs as a first-class integration risk, not a formatting detail. Standardize internally on the OpenAI delta shape, buffer tool-call fragments explicitly, read usage only from terminal events, and never parse partial JSON as complete. Use a normalization layer if you must support many models, but keep a thin provider-aware debug path so you can inspect what actually crossed the wire. Teams that ignore this ship agents that silently drop tool arguments and misbill tokens the first time they route to a non-OpenAI backend.