A beginner's guide to the OpenAI-compatible API standard

Most LLM infrastructure today speaks one dialect: the API that OpenAI shipped in 2023. This openai compatible api standard guide cuts through the marketing and shows the actual HTTP surface you need to target, the invariants you can rely on, and the sharp edges that break clients in production. If you build against the protocol instead of a single vendor, you keep the option to route around outages and price changes.

Why the OpenAI-compatible API exists

The shape of /v1/chat/completions became a de facto standard because every SDK, proxy, and eval harness coded against it. When a new model provider wants adoption, they mirror that schema. That convergence lets you swap base_url and api_key to talk to a different backend without rewriting business logic.

Compatibility is a spectrum, not a boolean. Some endpoints replicate only the chat endpoint; others add embeddings, audio, or fine-tune endpoints. Treat the standard as a contract for the request/response JSON, not a guarantee of feature parity. A provider that returns 200 OK with a different finish_reason vocabulary is still “compatible” enough to pass a smoke test and still break your parser.

The core endpoints you must know

POST /v1/chat/completions

The workhorse. Accepts model, messages, temperature, max_tokens, stream, and optional tools/functions. Returns id, object, created, model, choices, usage.

POST /v1/embeddings

Takes model and input (string or list). Returns vectors. Critical for retrieval pipelines, but note that dimension sizes and normalization differ across backends even when the route is identical.

GET /v1/models

Lists available model ids. Clients use this for capability discovery, but many gateways return an aggregated or static list that doesn’t reflect live availability.

A minimal chat call with the official Python client:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.openai.com/v1",
    api_key="sk-your-key",
)

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain rate limits"}],
)
print(resp.choices[0].message.content)

Swap base_url to any compatible gateway and the same code runs unchanged.

Anatomy of a request

The request body is JSON. Key invariants:

• messages is an array of {role, content}; roles are system, user, assistant, tool.
• model is a string opaque to the client; the server maps it to a backend.
• stream boolean triggers Server-Sent Events.

A raw curl shows the wire format exactly:

curl https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "ping"}],
    "max_tokens": 50
  }'

Non-streaming response:

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1699000000,
  "model": "gpt-4o-mini",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "pong"},
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15}
}

Handling streaming and tool calls

Set "stream": true and you get chunks:

{"id":"chatcmpl-123","object":"chat.completion.chunk","choices":[{"delta":{"content":"p"},"finish_reason":null}]}

Parse delta.content and concatenate. Watch for finish_reason: "tool_calls"; the delta then carries tool_calls partial JSON. Many clients mishandle incremental function-call arguments. Buffer and parse only when the stream closes.

Tradeoff: streaming reduces time-to-first-token but complicates error handling. A mid-stream HTTP 500 leaves you with partial output and no usage block. Design idempotent retries at the message level, not the token level. If you must bill per token, send stream_options: {"include_usage": true} and capture the final chunk’s usage.

Authentication and base URL swapping

Auth is Authorization: Bearer <key>. That’s the only header the standard mandates. Providers may add OpenAI-Organization or custom headers, but those break portability.

To point at a gateway, change two lines:

client = OpenAI(
    base_url="https://gateway.example/v1",
    api_key="gw-key",
)

If the gateway aggregates multiple providers, the model string becomes a routing directive. E.g., anthropic/claude-3-5-sonnet might route upstream. The openai compatible api standard guide assumes model names are opaque, so don’t hardcode prefixes in logic. Validate against the /v1/models response at startup and fail fast if a required capability is missing.

Common pitfalls when treating it as a strict standard

1. Parameter drift. max_tokens in OpenAI means completion tokens; some backends interpret it as total context. Set it conservatively and test with long prompts.
2. Missing fields. Not every implementation returns usage unless you send stream_options: {include_usage: true} while streaming.
3. Tool calling schema. OpenAI uses tools with function objects. Some compat servers only support legacy functions. Test both paths.
4. Temperature bounds. A few open-source models clip at 1.0; others accept 2.0. Clamp client-side to avoid silent truncation.
5. Content types. Always send Content-Type: application/json. Some proxies reject form-encoded or missing charset.
6. Embedding normalization. One backend returns unit vectors; another returns raw. Cosine similarity still works, but dot product billing may surprise you.

These deviations mean you should write a thin adapter that normalizes responses before they hit your app. Don’t let domain code know which provider answered.

Routing, fallbacks, and cache control

In production you rarely call one provider. A gateway that speaks the protocol can abstract failover. For example, n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models and performs automatic fallback when a provider is rate-limited or degraded, while honoring client routing directives and forwarding provider cache-control hints. Your client sends the same request; the gateway retries upstream or shifts models based on x-routing headers.

Cache control is subtle. OpenAI accepts cache_control inside system messages for prompt caching. Compatible gateways must forward those hints or you pay full token cost. Verify the gateway passes them through by inspecting response headers or billing line items. If a gateway strips cache_control, precompute and cache embeddings client-side instead.

A practical migration path

Follow this ordered path to adopt the standard without locking yourself in:

1. Pin the SDK. Use the official openai Python/TS package; it parses the schema strictly and surfaces deprecations.
2. Extract base URL and key into env vars. Never hardcode.
3. Write a normalization wrapper. Map usage to your internal telemetry; default missing fields like system_fingerprint.
4. Implement stream-safe retries. On connection reset, replay the full message array, not the partial stream.
5. Add a model registry. Store model id → capability map (context window, supports tools) externally.
6. Test against a mock. Stand up a local server that returns canned chat/embeddings responses to validate your adapter.
7. Switch base_url to a gateway and run shadow traffic before cutover. Compare token counts and latency distributions.

Testing your client against multiple backends

Build a matrix test that runs the same request against each configured backend:

import pytest
from openai import OpenAI

BACKENDS = [
    ("https://api.openai.com/v1", "sk-openai"),
    ("https://gateway.example/v1", "gw-key"),
]

@pytest.mark.parametrize("base,key", BACKENDS)
def test_chat(base, key):
    c = OpenAI(base_url=base, api_key=key)
    r = c.chat.completions.create(
        model="test-model",
        messages=[{"role": "user", "content": "hi"}],
        max_tokens=5,
    )
    assert r.choices[0].message.content

Run it in CI against a stub to catch schema drift. When a provider changes a field, your test fails before users see it. Extend the matrix to embeddings and tool calls; those are where silent incompatibility hides.

The openai compatible api standard guide above gives you the minimum viable mental model. Implement the adapter, treat model names as opaque routing tokens, and keep streaming errors isolated. That’s the difference between a demo and a system that survives a provider outage.