OpenAI-compatible streaming compatibility describes the contract where a non-OpenAI inference endpoint emits Server-Sent Events (SSE) shaped like OpenAI’s chat/completions streaming responses, so the official SDK or any client built to that wire format can consume it without code changes. It is a wire-format and semantic convention, not a guarantee of model behavior parity. If you are debugging a gateway or rolling your own proxy, understanding exactly which parts of the stream are standardized and which are conventionally faked will save you hours.
What the stream wire format specifies
The OpenAI streaming response is plain HTTP chunked transfer with text/event-stream. Each event is a line data: {json}\n\n. The JSON mirrors a non-streaming chat.completion object but with choices[].delta instead of choices[].message. A terminal event is data: [DONE].
{
"id": "chatcmpl-123",
"object": "chat.completion.chunk",
"created": 1690000000,
"model": "gpt-4o-mini",
"choices": [
{
"index": 0,
"delta": {"role": "assistant", "content": "Hello"},
"finish_reason": null
}
]
}The first chunk typically carries delta.role. Subsequent chunks carry delta.content fragments. Tool calls arrive as delta.tool_calls with an index and incremental function.arguments strings. The stream ends when finish_reason is set on the final delta and the [DONE] sentinel is sent.
Why OpenAI-compatible streaming compatibility matters
Engineers adopt this contract to avoid vendoring multiple SDKs. A single OpenAI client instance pointed at a different base_url should work against Anthropic, Mistral, or a local vLLM server if the endpoint honors the shape. In a multi-provider routing layer, this compatibility is the only thing that makes transparent fallback possible. A gateway like n4n.ai that exposes one OpenAI-compatible endpoint across 240+ models leans on this contract so existing SDK code just works; automatic fallback when a provider is degraded stays invisible to the caller because the stream shape is preserved.
Without it, you rewrite streaming parsers per backend, handle divergent error frames, and ship subtle bugs when a model emits a partial JSON tool call.
How a minimal client consumes it
The OpenAI Python SDK handles SSE parsing, delta assembly, and [DONE] termination. Point it at any compatible base URL:
from openai import OpenAI
client = OpenAI(base_url="https://api.example.com/v1", api_key="sk-...")
stream = client.chat.completions.create(
model="mistral-7b",
messages=[{"role": "user", "content": "Explain SSE in one line"}],
stream=True,
stream_options={"include_usage": True},
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
if chunk.usage:
print(f"\nTokens: {chunk.usage.total_tokens}")The same code runs against OpenAI, a self-hosted server, or a gateway. That is the entire value proposition.
What breaks in practice
Usage metadata is optional and often wrong
OpenAI only sends usage in the final chunk when you pass stream_options={"include_usage": true}. Many compatible servers either omit it, send 0, or send it on every chunk. If your billing or logging depends on chunk.usage, verify the field exists and is populated. A gateway that does per-token usage metering must reconcile provider-reported counts with its own tokenization because upstream numbers are unreliable.
Tool call streaming fragmentation
OpenAI streams tool_calls as a sequence of deltas with the same index. The id and type appear once; function.name may arrive whole or split; function.arguments is a concatenated JSON string across chunks. Naive clients that parse arguments per-chunk will throw. You must accumulate the string and json.loads only after finish_reason == "tool_calls".
acc = {}
for chunk in stream:
tc = chunk.choices[0].delta.tool_calls
if tc:
for call in tc:
acc.setdefault(call.index, {"name": "", "args": ""})
if call.function.name:
acc[call.index]["name"] += call.function.name
if call.function.arguments:
acc[call.index]["args"] += call.function.argumentsPing frames and comment lines
OpenAI’s reference implementation does not send periodic pings, but proxies behind nginx or load balancers often inject : ping\n\n comment lines to keep connections alive. The official SDK ignores lines not starting with data:. A hand-rolled split('\n') parser that assumes every non-empty line is JSON will crash. Strip comments explicitly.
Mid-stream errors
A compliant stream starts with HTTP 200. If the model OOMs halfway, OpenAI sends a final data: {"error": ...} event before [DONE]. Some compat servers instead reset the TCP connection, causing the SDK to raise ChunkedDecodeError. Your code must distinguish a clean finish_reason from a truncated stream. Implement a read timeout and treat connection closure without [DONE] as a retryable failure.
Role and content nullability
The first delta may have role: "assistant" and empty content. Later deltas have content but no role. Some implementations send role on every chunk, others never send content on the first. Do not assert delta.role exists on every chunk.
Concrete debugging example
When a stream silently truncates, drop to curl to inspect raw bytes:
curl -N https://api.example.com/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-H "Content-Type: application/json" \
-d '{"model":"test","messages":[{"role":"user","content":"hi"}],"stream":true}' \
| grep -a '^data:' | head -30Look for: missing data: prefix, raw JSON without newline framing, or a sudden EOF. If you see data: [DONE] absent, the server closed early. If you see data: {"error":...} after content, the client SDK should have raised but a custom parser may have swallowed it.
Common misconceptions
“Compatible” means the model behaves like GPT-4
It does not. The contract covers bytes on the wire, not reasoning quality, token vocabulary, or tool-calling semantics. A model that streams valid chunks can still emit malformed tool arguments or refuse the task. Compatibility is necessary but insufficient for drop-in replacement.
The OpenAI SDK is a strict validator
The SDK is lenient. It ignores unknown fields, tolerates missing usage, and silently drops ping comments. This leniency hides incompatibilities until you write a strict consumer or a different language binding (e.g., a Rust or Go client) that validates the schema.
Streaming implies lower latency end-to-end
Time-to-first-token depends on queueing and prefill, not the SSE format. A compatible endpoint can have worse tail latency than OpenAI because it fronts a slower provider. The format does not magically speed up inference.
All providers honor stream_options
Several open-weight servers ignore include_usage entirely. If you need token counts, check the raw response or compute locally with a tokenizer matching the model.
Verification checklist
Before declaring an endpoint OpenAI-compatible streaming compatibility complete, test these:
- Stream starts with HTTP 200 and
text/event-stream. - Every event is
data:prefixed, double-newline terminated. [DONE]sentinel sent exactly once at end.choices[0].deltapresent on all chunks;choices[0].finish_reasonset on final delta.usageappears only when requested, and values are non-zero if tokens were produced.- Tool calls stream with stable
index, accumulate to valid JSON. - Connection stays open with comment pings if longer than 30s without tokens.
- Mid-stream failure emits an error event or cleanly closes with recognizable signal.
Run this matrix against each model behind your gateway. The ones that fail are not “OpenAI-compatible” no matter what the marketing says.
Closing note on routing directives
Clients can send headers or body extensions to force a provider or disable cache. A correct gateway forwards provider cache-control hints and honors routing directives while still normalizing the stream to the OpenAI shape. That normalization is where most breakage is hidden: the upstream may speak a different protocol entirely, and the translation layer is the only thing standing between your SDK and a 500. Test the translation, not just the model.