How to swap OpenAI's base_url for a unified LLM gateway

When you swap openai base url for gateway, you re-point the OpenAI SDK at an OpenAI-compatible endpoint that proxies multiple providers behind one contract. The code change is usually a one-line diff, but the surrounding config, model identifiers, and header passthrough deserve attention so you don’t break caching or usage accounting. This guide gives an end-to-end migration path using the official Python and TypeScript SDKs.

Step 1: Inventory every client instantiation

Before editing anything, find where your code constructs an OpenAI client. In a Python service it typically looks like:

from openai import OpenAI
import os

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

TypeScript is nearly identical:

import OpenAI from "openai";
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

Run a grep across the repo for new OpenAI, OpenAI(, and api.openai.com. Note any custom base_url already set—some teams point at Azure OpenAI or an internal proxy. List each call site; you will either edit the constructor directly or inject an environment variable. In a monorepo, a shared llm_client.py module is the best place to centralize the change. If the client is buried in a lambda or a cron job, capture those separately.

Step 2: Choose the gateway endpoint and key

A unified gateway exposes a single OpenAI-compatible REST surface. For example, n4n.ai provides an endpoint at https://api.n4n.ai/v1 that fronts 240+ models behind the same /chat/completions contract you already use. Sign up, create a gateway key scoped to your project, and store it as GATEWAY_API_KEY in your secret manager.

Do not hardcode the key. The gateway authenticates with its own token, not your OpenAI token. If you already have OPENAI_API_KEY in env, rename or override it to avoid confusion. Treat the gateway key like any other production secret: rotate it, log only the last four characters, and restrict it by IP if the gateway supports that.

Step 3: Swap the base_url in code

The minimal change is passing base_url to the constructor. Python:

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ["GATEWAY_API_KEY"],
    base_url="https://api.n4n.ai/v1",  # swap openai base url for gateway here
)

TypeScript:

import OpenAI from "openai";
const client = new OpenAI({
  apiKey: process.env.GATEWAY_API_KEY,
  baseURL: "https://api.n4n.ai/v1",
});

If you prefer zero code changes, the OpenAI SDK respects base_url via the OPENAI_BASE_URL environment variable in most deployments. Set:

export OPENAI_BASE_URL="https://api.n4n.ai/v1"
export OPENAI_API_KEY="$GATEWAY_API_KEY"

Then existing OpenAI() calls without an explicit base_url will hit the gateway. Verify your SDK version actually reads that env var; the official v1.x client does when base_url is omitted. Watch for a trailing slash—most gateways reject https://api.n4n.ai/v1/ with a 404 on the joined path.

Step 4: Reconcile model identifiers

OpenAI SDK calls specify model="gpt-4o". Gateways may accept that verbatim or require a provider prefix like openai/gpt-4o or anthropic/claude-3-5-sonnet. Check the gateway’s model list. If prefixes are required, centralize the mapping so business logic stays clean:

MODEL_MAP = {
    "gpt-4o": "openai/gpt-4o",
    "claude": "anthropic/claude-3-5-sonnet",
    "mistral": "mistralai/mistral-large",
}

def complete(prompt: str, logical_model: str):
    resp = client.chat.completions.create(
        model=MODEL_MAP[logical_model],
        messages=[{"role": "user", "content": prompt}],
    )
    return resp.choices[0].message.content

This keeps your product code referencing gpt-4o while the gateway routes to the correct backend. If you later add a fallback model, extend the map rather than touching call sites.

Step 5: Preserve cache-control and routing hints

OpenAI’s prompt caching uses beta headers or extra_body fields. When you swap openai base url for gateway, those hints must traverse the proxy unchanged. The SDK sends them via extra_headers or extra_body:

resp = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": long_system_prompt}],
    extra_headers={"cache-control": "max-age=300"},
)

A gateway that forwards provider cache-control hints passes this to the upstream provider without stripping it. Similarly, if you need to pin a provider or request fallback behavior, pass routing directives as headers:

extra_headers={"x-gateway-route": "fallback"}

The gateway honors client routing directives, so your fallback logic stays in code rather than infrastructure config. Confirm with a request trace that the header reaches the gateway and the upstream call includes the cache flag.

Step 6: Use automatic fallback and per-token metering

One concrete benefit of the gateway is automatic fallback when a provider is rate-limited or degraded. You no longer need custom retry loops for upstream 429s; the gateway returns a normal completion from a secondary provider if you’ve configured that policy. Your error handling should still catch APIStatusError, but the frequency drops.

Usage metering is per-token and appears in the standard usage object:

print(resp.usage.prompt_tokens, resp.usage.completion_tokens)

Because the gateway emits OpenAI-compatible usage, existing billing or logging code works without modification. If you export metrics to Prometheus, scrape prompt_tokens and completion_tokens exactly as before. The per-token usage metering gives the same granularity you had with OpenAI directly, which matters when you mix expensive and cheap models behind one client.

Step 7: Verify the migration

Write a throwaway script and run it against the gateway:

from openai import OpenAI
import os, time

client = OpenAI(api_key=os.environ["GATEWAY_API_KEY"], base_url="https://api.n4n.ai/v1")
start = time.time()
resp = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Say hello in 5 words."}],
)
print(resp.choices[0].message.content)
print("usage:", resp.usage.model_dump())
print("latency_ms:", int((time.time()-start)*1000))

Success criteria:

• Response parses as a valid ChatCompletion.
• usage.prompt_tokens > 0 and completion_tokens > 0.
• Latency is within your SLO (typically < 2s for tiny prompts).
• resp.model matches the routed model string.

You can also curl the raw endpoint to confirm the protocol layer:

curl https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"openai/gpt-4o","messages":[{"role":"user","content":"hi"}]}'

A 200 with choices and usage confirms the swap openai base url for gateway worked at the HTTP level. Add a pytest smoke test that runs in CI against a gateway staging key to catch regressions.

Step 8: Roll out without surprises

Don’t flip all traffic at once. Use an env var and a feature flag:

import os
from openai import OpenAI

if os.environ.get("USE_GATEWAY"):
    client = OpenAI(api_key=os.environ["GATEWAY_API_KEY"], base_url="https://api.n4n.ai/v1")
else:
    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

Run the gateway path in shadow mode: send duplicates to both, compare outputs for a sample of requests. Once error rates and p95 latencies match, switch the flag for real users. If you use a load balancer or sidecar, you can also redirect at the network layer and keep application code untouched.

Common pitfalls

Streaming: The SDK’s stream=True works unchanged, but verify the gateway passes through SSE frames without buffering. Test an explicit streaming loop.

Async clients: AsyncOpenAI takes the same base_url argument. Audit both sync and async paths; missing one causes inconsistent behavior.

Timeouts: Gateways add a few milliseconds of routing overhead. If you set a tight 5s timeout, bump it to 10s during the transition.

Capability gaps: Not every model supports response_format or tools. The gateway returns the upstream error; handle it as you would with OpenAI.

Key confusion: A common outage cause is pointing base_url at the gateway but leaving OPENAI_API_KEY as the OpenAI key. The gateway rejects it with 401.

Verification checklist

• All OpenAI() / new OpenAI() constructors use the gateway key and base_url.
• Model names resolve through the gateway (prefixed or passthrough).
• Cache-control headers appear in outbound requests (inspect gateway logs).
• Usage tokens logged and match expectations per request.
• Fallback tested by temporarily blocking the primary provider if the gateway allows.
• CI runs a smoke test against a gateway staging endpoint.
• Shadow mode compared outputs for at least 100 representative prompts.

Swapping the base URL is trivial; the real work is confirming the surrounding contract holds. Do that, and you get multi-model access with zero changes to your completion logic.