Drop-in replacement checklist for OpenAI-compatible gateways

Swapping your LLM provider shouldn’t require rewriting application logic. This drop-in replacement checklist OpenAI-compatible gateway walks through the exact steps to point an existing OpenAI SDK client at a new backend, validate behavior, and handle edge cases that break naive migrations.

Step 1: Audit your current OpenAI SDK usage

Before changing any configuration, capture every assumption your code makes about the OpenAI API. List the models you call, the parameters you pass, and whether you use streaming, function tools, or response formatting.

# Example: inventory script for a Python service
import inspect
from your_app import llm_calls  # hypothetical registry

for call in llm_calls:
    print(call.model, call.temperature, call.stream, call.tools)

Check for hardcoded base_url overrides, custom headers, and timeout settings. The drop-in replacement checklist OpenAI-compatible gateway requires that the new endpoint supports the same surface area.

Verification

Run your inventory and confirm you have a complete list of model IDs and features. If you use json_mode or seed, note them explicitly.

Step 2: Repoint the base URL and credentials

The OpenAI SDK reads base_url and api_key from the client constructor. Swap these values via environment variables to avoid code changes.

from openai import OpenAI
import os

client = OpenAI(
    base_url=os.environ["LLM_GATEWAY_BASE_URL"],
    api_key=os.environ["LLM_GATEWAY_API_KEY"],
)

For Node.js:

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: process.env.LLM_GATEWAY_BASE_URL,
  apiKey: process.env.LLM_GATEWAY_API_KEY,
});

Set the env vars to the gateway endpoint. For example, a gateway such as n4n.ai exposes a single OpenAI-compatible endpoint covering 240+ models, so you only change the URL.

Verification

Run a minimal completion:

resp = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "ping"}],
)
print(resp.choices[0].message.content)

A non-200 response or schema mismatch means the gateway is not fully compatible.

Step 3: Map model identifiers

Model names differ across providers. Gateways often alias popular models to their own catalog. Build a translation table.

{
  "gpt-4o": "openai/gpt-4o",
  "claude-3-opus": "anthropic/claude-3-opus",
  "mixtral-8x7b": "mistral/mixtral-8x7b"
}

If your code references model directly, inject the mapping at the client boundary:

MODEL_MAP = {"gpt-4o": "openai/gpt-4o"}

def chat(model, messages):
    return client.chat.completions.create(
        model=MODEL_MAP.get(model, model),
        messages=messages,
    )

Verification

Call each mapped model with a trivial prompt. Confirm the response object shape matches the OpenAI SDK’s typed responses.

Step 4: Handle authentication headers and cache control

Some gateways require extra headers for routing or billing. The OpenAI SDK allows default_headers.

client = OpenAI(
    base_url=os.environ["LLM_GATEWAY_BASE_URL"],
    api_key=os.environ["LLM_GATEWAY_API_KEY"],
    default_headers={"X-Route-Preference": "cost-optimized"},
)

Provider cache-control hints (e.g., cache_control in Anthropic) must be forwarded. A compliant gateway honors client routing directives and forwards provider cache-control hints without stripping them.

Verification

Inspect the outgoing request with a proxy or the gateway’s debug log. Ensure headers arrive intact and the cache hint appears in the upstream call.

Step 5: Validate request and response parity

Write a differential test that sends identical requests to the old and new endpoints and compares outputs structurally.

import json

def serialize(resp):
    return {
        "id": resp.id,
        "choices": [c.message.content for c in resp.choices],
        "usage": resp.usage.model_dump(),
    }

old_resp = old_client.chat.completions.create(**payload)
new_resp = client.chat.completions.create(**payload)

assert serialize(old_resp)["choices"]  # structure check
assert new_resp.usage.prompt_tokens > 0

Focus on usage fields, finish_reason, and streaming chunks.

Verification

Run the test across all models in your audit. Any missing field is a blocker.

Step 6: Test streaming and tool calls

Streaming is where many gateways diverge. Use the SDK’s stream iterator:

stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "count to 3"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

For tools:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {"type": "object", "properties": {"city": {"type": "string"}}},
    },
}]
resp = client.chat.completions.create(model="gpt-4o", messages=[...], tools=tools)
print(resp.choices[0].message.tool_calls)

Verification

Confirm token deltas sum to the final usage.completion_tokens. For tools, ensure tool_calls is populated with correct arguments.

Step 7: Implement fallback and routing logic

Providers degrade. A robust drop-in replacement checklist OpenAI-compatible gateway includes fallback behavior. If the gateway doesn’t auto-fallback, implement a retry with a secondary model.

from openai import APIStatusError

def chat_with_fallback(model, messages):
    try:
        return client.chat.completions.create(model=model, messages=messages)
    except APIStatusError as e:
        if e.status_code == 429:
            return client.chat.completions.create(model="backup-model", messages=messages)
        raise

Some gateways (e.g., n4n.ai) provide automatic fallback when a provider is rate-limited or degraded, which removes the need for client-side retry logic.

Verification

Simulate a 429 from the primary model (via fault injection) and confirm the backup responds or the gateway fails over transparently.

Step 8: Monitor per-token usage metering

Billing depends on accurate token counts. Pull usage from each response and ship it to your metrics pipeline.

resp = client.chat.completions.create(...)
metrics.record(
    model=resp.model,
    prompt_tokens=resp.usage.prompt_tokens,
    completion_tokens=resp.usage.completion_tokens,
)

Gateways with per-token usage metering let you reconcile bills without custom instrumentation.

Verification

Compare metered tokens against your own tokenizer estimate for a sample of requests. Discrepancies above 2% warrant investigation.

Step 9: Cutover and rollback

Flip the environment variable for a canary percentage of traffic. Keep the old client instantiated but unused.

if feature_flag.enabled("new_gateway"):
    active_client = client
else:
    active_client = old_client

Verification

Watch error rates and latency for 24 hours. If p99 latency doubles or error rate exceeds 1%, revert by toggling the flag.

Final checklist

Following this drop-in replacement checklist OpenAI-compatible gateway approach lets you migrate without touching business logic. The key is treating the API as a contract and verifying each field.