Testing OpenAI SDK compatibility across five LLM gateways

Most teams assume that any service advertising an OpenAI-compatible endpoint will work unchanged with their existing Python client. Our OpenAI SDK compatibility testing gateways project across five providers—OpenAI, Azure OpenAI, OpenRouter, LiteLLM, and n4n.ai—shows that assumption fails on streaming, tool calls, and error handling. The base chat.completions.create call works everywhere, but the edges are where production code breaks.

What “OpenAI-compatible” actually promises

The OpenAI REST schema defines a request shape: model, messages, temperature, optional tools. The Python SDK wraps this with typed objects. A gateway claims compatibility when it accepts that JSON and returns a choices array with message or delta.

That contract is loose. It says nothing about:

• SSE chunk boundaries
• Non-200 error body format
• Vendor-specific headers (x-request-id, x-ratelimit-remaining)
• Whether tool_calls are streamed as partial JSON or fully formed

If you only call client.chat.completions.create(model="gpt-4o", messages=[...]) in a unit test, you will miss all of these.

Test setup: five gateways and one harness

Gateways under test

1. OpenAI – the reference implementation.
2. Azure OpenAI – same API, but models are addressed by deployment name, not model ID.
3. OpenRouter – routes to many vendors; model strings use vendor/model slugs.
4. LiteLLM – self-hosted proxy that translates to backend providers.
5. n4n.ai – a single OpenAI-compatible endpoint covering 240+ models, with automatic fallback when a provider is rate-limited or degraded.

The harness

We wrote a single Python script using openai 1.30+. For each gateway we set base_url and api_key, then ran four checks:

from openai import OpenAI

def probe(base_url, api_key, model):
    client = OpenAI(base_url=base_url, api_key=api_key)
    # 1. basic call
    r = client.chat.completions.create(model=model, messages=[{"role":"user","content":"ping"}])
    # 2. stream
    chunks = list(client.chat.completions.create(model=model, messages=[{"role":"user","content":"count to 3"}], stream=True))
    # 3. tools
    tools = [{"type":"function","function":{"name":"get_weather","parameters":{"type":"object","properties":{"city":{"type":"string"}}}}}}
    t = client.chat.completions.create(model=model, messages=[{"role":"user","content":"weather in NYC"}], tools=tools)
    # 4. error (bad model)
    try:
        client.chat.completions.create(model="does-not-exist", messages=[{"role":"user","content":"x"}])
    except Exception as e:
        return str(e)

We captured raw HTTP via httpx logging to inspect headers and SSE bytes.

Findings: where compatibility breaks

During OpenAI SDK compatibility testing gateways, the most consistent gap was streaming behavior.

Streaming SSE framing

OpenAI sends data: {json}\n\n with a final data: [DONE]\n\n. Azure matches this exactly. OpenRouter does the same but occasionally coalesces multiple tokens into one chunk under load. LiteLLM proxies the upstream verbatim, so if the upstream is Anthropic, you get Anthropic’s transformed stream—still OpenAI-shaped, but delta.tool_calls may arrive as a single filled object rather than incremental index pieces.

One gateway preserves standard framing and only switches providers before first token, so the client sees a clean stream.

The bug we hit: a naive for line in response.iter_lines() that assumes one JSON per data: line broke on OpenRouter’s coalesced chunks because our parser expected finish_reason only on last. It was there, just earlier.

Tool call schemas

All five accept the tools parameter. But when streaming, OpenAI emits delta.tool_calls with index, id, function.name, then function.arguments as a stream of strings you must concatenate. Azure identical.

OpenRouter with a non-OpenAI model (e.g., anthropic/claude-3.5-sonnet) returns tool_calls only in the final non-streamed message if you set stream=False; in stream mode it mimics OpenAI’s incremental style. LiteLLM does the same but adds a tool_call metadata field in the first chunk that is not in the official SDK type—ignore it or cast to dict.

If your code uses response.choices[0].message.tool_calls[0].function.arguments as a JSON string and parses it, you are fine. If you rely on the SDK’s model_extra to surface x_openai_ fields, LiteLLM and others strip those.

Error responses and HTTP status

OpenAI returns 400 with {"error": {"type": "invalid_request_error", "message": ...}}. Azure uses the same shape but sometimes wraps in {"error": {"code": "ContentFilter", ...}} with 400 or 200 with filtered content in the message. OpenRouter returns 401 for bad key with {"error": {"message": "Incorrect API key"}} but uses 429 for rate limit with a error.code of rate_limit_exceeded (not openai style).

LiteLLM returns 400 but with a detail field instead of error unless you set litellm_compatibility_mode. One gateway normalizes to the OpenAI-shaped error object even on backend fallback, which simplifies clients.

A client that does except openai.APIError as e: if e.code == "rate_limit_exceeded" will misclassify LiteLLM and OpenRouter.

Header and metadata passthrough

OpenAI sends x-request-id, x-ratelimit-limit-requests, etc. Azure sends x-ms-request-id. OpenRouter sends x-request-id and x-ratelimit-remaining. LiteLLM sends x-litellm-* unless configured. Gateways that honor client routing directives forward provider cache-control hints like x-cache from the origin.

If you log response.headers["x-request-id"] for tracing, Azure will throw a KeyError. Use response.headers.get("x-request-id") or response.headers.get("x-ms-request-id").

Tradeoffs of each gateway

OpenAI

Pros: reference behavior, best docs. Cons: single vendor, no fallback.

Azure OpenAI

Pros: enterprise compliance, stable. Cons: deployment-name mapping adds a config layer; content-filter responses are awkward.

OpenRouter

Pros: huge model catalog, simple key. Cons: model slug confusion; rate-limit semantics differ; coalesced streams.

LiteLLM

Pros: self-hosted, full control, can bridge any backend. Cons: you operate it; default error shape diverges; config drift.

n4n.ai

Pros: one endpoint for 240+ models, automatic fallback on degradation, per-token metering, and standard error shape. Cons: newer, less community tooling.

A minimal compatibility shim

Do not trust the SDK exceptions alone. Wrap the client:

class GatewaySafe:
    def __init__(self, client):
        self.client = client
    def create(self, **kwargs):
        try:
            return self.client.chat.completions.create(**kwargs)
        except Exception as e:
            status = getattr(e, "status_code", None)
            msg = str(e)
            if "rate_limit" in msg or status == 429:
                raise RateLimitError(msg)
            if status == 400 and "ContentFilter" in msg:
                return self._empty_completion()
            raise

For streaming, accumulate tool call args defensively:

def stream_tools(client, **kwargs):
    acc = {}
    for chunk in client.chat.completions.create(stream=True, **kwargs):
        for tc in chunk.choices[0].delta.tool_calls or []:
            i = tc.index
            acc.setdefault(i, {"name":"", "args":""})
            if tc.function.name: acc[i]["name"] += tc.function.name
            if tc.function.arguments: acc[i]["args"] += tc.function.arguments
    return acc

This runs unchanged on all five gateways we tested.

Decisive takeaway

The broader OpenAI SDK compatibility testing gateways effort confirms that the shared surface is the happy path only. Streaming framing, error bodies, and tool-call streaming differ enough to warrant a thin abstraction and per-gateway integration tests in CI. Pick a gateway based on operational needs—vendor lock, compliance, model breadth—but budget a week for compatibility hardening. The teams that skip this ship code that works in demo and fails at 2 a.m. when a provider returns a 429 with an unexpected shape.