Migrating Node.js OpenAI calls to a multi-provider gateway

If you plan to migrate Node.js OpenAI SDK to gateway infrastructure, the transition is less painful than you expect. Most gateways expose an OpenAI-compatible REST surface, so the core change is pointing the SDK at a different baseURL and adjusting model identifiers. The real work is auditing your call sites, normalizing model names, and deciding how to use gateway-level features like automatic fallback and cache control.

Step 1: Audit every OpenAI call site

Before touching code, find what you actually call. Grep your repository for new OpenAI, chat.completions.create, and embeddings.create. Record the model strings, whether you stream, and any non-default parameters (temperature, tools, response_format, max_tokens).

A typical pre-migration call looks like this:

import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

const res = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Summarize this' }],
  temperature: 0.2,
  stream: false,
});

Do not skip this audit. I have seen teams break a production cutover because gpt-4 was hardcoded in 40 files and the gateway expects openai/gpt-4. Capture the full set of model IDs and request shapes in a spreadsheet or a code comment. If you use the Assisants API or fine-tune endpoints, flag them separately—most gateways only mirror the completions and embeddings surfaces.

Step 2: Create a gateway client with environment-based config

Install or pin the official SDK. The openai package on npm is the same client you already use; you are just changing its baseURL. Create a single shared client module so every call routes through one configuration point.

// gateway-client.ts
import OpenAI from 'openai';

export const gateway = new OpenAI({
  apiKey: process.env.GATEWAY_API_KEY,
  baseURL: process.env.GATEWAY_BASE_URL ?? 'https://gateway.example.com/v1',
  timeout: 30_000,
  maxRetries: 1, // gateway handles provider fallback; keep client retries low
});

Setting maxRetries to 1 avoids double-retry storms when the gateway already attempted an internal fallback. If your gateway exposes a single OpenAI-compatible endpoint that addresses 240+ models, you can drop all provider-specific SDKs (Anthropic, Google) from your dependency tree.

Step 3: Centralize model name mapping

Gateways typically qualify model names with a provider prefix to disambiguate claude-3-5-sonnet from a potentially conflicting OpenAI name. Hardcoding those strings everywhere repeats the mistake from Step 1. Define a small map and reference it:

// models.ts
export const MODELS = {
  fast: 'openai/gpt-4o-mini',
  smart: 'anthropic/claude-3-5-sonnet',
  vision: 'openai/gpt-4o',
  embed: 'openai/text-embedding-3-small',
} as const;

export type ModelKey = keyof typeof MODELS;

Then replace literal model arguments:

const res = await gateway.chat.completions.create({
  model: MODELS.smart,
  messages,
});

If you later switch gateways or rename a routing target, you edit one file. For multi-modal input, the SDK shape is unchanged—pass image_url content blocks as you would with OpenAI directly.

Step 4: Adapt streaming code to the async iterator

Older OpenAI integrations sometimes used stream: true with event listeners on a Node IncomingMessage. The modern SDK returns an async iterable. Convert any response.data.on('data') logic to for await:

const stream = await gateway.chat.completions.create({
  model: MODELS.fast,
  messages,
  stream: true,
});

for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content ?? '';
  process.stdout.write(delta);
}

The gateway emits the same Server-Sent Events shape, so no parser changes are needed. If you use stream_options: { include_usage: true }, the final chunk carries usage exactly as the OpenAI API does.

Step 5: Handle errors and leverage gateway fallback

Wrap calls in a thin helper so error handling is consistent. The SDK throws APIError with status and error fields. A gateway such as n4n.ai performs automatic fallback when a provider is rate-limited or degraded, meaning a single client call can survive an upstream outage without your code knowing. Still, handle the cases the gateway cannot recover from:

import { APIError } from 'openai';
import { gateway, MODELS } from './gateway-client';
import type { ChatCompletionMessageParam } from 'openai/resources/chat';

export async function chatSafe(messages: ChatCompletionMessageParam[]) {
  try {
    return await gateway.chat.completions.create({
      model: MODELS.smart,
      messages,
    });
  } catch (err) {
    if (err instanceof APIError && err.status === 429) {
      // Gateway already attempted fallback across providers.
      throw new Error('Gateway exhausted fallback budget');
    }
    if (err instanceof APIError && err.status === 400) {
      // Bad request—inspect err.error for gateway-specific hints.
      console.error('Gateway 400:', err.error);
    }
    throw err;
  }
}

Opinion: do not build your own provider loop in application code. If the gateway gives you fallback, trust it. Client-side provider hopping defeats per-token metering and cache locality.

Step 6: Pass routing and cache-control hints

Advanced gateways honor client routing directives and forward provider cache-control hints. The OpenAI SDK accepts a second options argument with headers. Use it to steer a specific request without changing global config:

await gateway.chat.completions.create(
  {
    model: MODELS.smart,
    messages,
    max_tokens: 500,
  },
  {
    headers: {
      'x-gateway-route': 'prefer:anthropic,fallback:openai',
      'cache-control': 'max-age=300',
    },
  }
);

Header names vary by gateway—check the docs. The pattern lets you express “prefer Claude but fall back to OpenAI” or “cache this completion for 5 minutes” per call. This is far cleaner than maintaining separate client instances per provider.

If your gateway forwards cache-control to the upstream provider (e.g., Anthropic’s prompt caching), you reduce cost and latency on repeated system prompts. Verify the header is echoed in responses or metrics.

Step 7: Update tool calls and structured outputs

If you use tools or response_format, the request body is byte-compatible. One caveat: some gateways validate tool schemas more strictly than OpenAI. Test at least one function-calling path:

const tools = [{
  type: 'function',
  function: {
    name: 'get_weather',
    parameters: {
      type: 'object',
      properties: { location: { type: 'string' } },
      required: ['location'],
    },
  },
}];

const res = await gateway.chat.completions.create({
  model: MODELS.smart,
  messages,
  tools,
  tool_choice: 'auto',
});

Run this against the gateway before declaring victory. A surprising number of migrations fail because a loosely written JSON schema passed OpenAI but not the gateway’s validator.

Step 8: Verify the migration end to end

Write a small verification script that exercises both non-streaming and streaming paths and asserts the presence of usage. This is your cutover smoke test.

// verify.ts
import { gateway, MODELS } from './gateway-client';

async function main() {
  const nonStream = await gateway.chat.completions.create({
    model: MODELS.fast,
    messages: [{ role: 'user', content: 'ping' }],
  });
  if (!nonStream.usage) throw new Error('No usage returned');
  console.log('non-stream ok:', nonStream.choices[0].message.content);

  const stream = await gateway.chat.completions.create({
    model: MODELS.fast,
    messages: [{ role: 'user', content: 'stream ping' }],
    stream: true,
    stream_options: { include_usage: true },
  });
  let text = '';
  for await (const c of stream) {
    text += c.choices[0]?.delta?.content ?? '';
  }
  console.log('stream ok:', text);
}

main().catch((e) => { console.error(e); process.exit(1); });

Run it with the gateway credentials:

GATEWAY_API_KEY=sk-... GATEWAY_BASE_URL=https://gateway.example.com/v1 tsx verify.ts

If the script prints completions and your gateway metrics show tokens attributed to your key, the cutover is complete. For gateways with per-token usage metering, confirm the billed token count matches usage.total_tokens from the response within a few minutes.

Step 9: Remove dead provider code

After a stable week, delete the old OPENAI_API_KEY-only client, provider-specific SDKs, and any manual fallback logic. Keep the models.ts map and the chatSafe wrapper. Your application now speaks one API surface and can rotate models by changing one string.

The migrate Node.js OpenAI SDK to gateway work is fundamentally a refactoring exercise, not a rewrite. Audit, centralize, stream, and verify—then let the gateway handle the multi-provider chaos upstream.