Multi-model fallback in Node.js with the OpenAI SDK

Building resilient LLM integrations means surviving rate limits, provider incidents, and model-specific errors. This tutorial implements node.js openai sdk multi-model fallback so a single failed model doesn’t take down your request path. You’ll walk away with a drop-in function that tries a ranked list of models and only errors when every option is exhausted.

Prerequisites

• Node.js 20+ (for native fetch, AbortController, and AbortSignal.timeout).
• An API key from any OpenAI-compatible provider. Set it as OPENAI_API_KEY in a .env file.
• The OpenAI Node SDK v4: npm install openai dotenv.
• Basic TypeScript ESM knowledge. We’ll run with tsx.

Create the project scaffold:

mkdir fallback-demo && cd fallback-demo
npm init -y
npm pkg set type=module
npm install openai dotenv
npm install -D tsx

Setting up the client

The OpenAI SDK accepts a baseURL, so it works against any compatible gateway or local mock. We point it at OpenAI here, but the same code runs against other endpoints.

// client.ts
import OpenAI from 'openai';
import 'dotenv/config';

export const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY!,
  // baseURL: 'https://api.openai.com/v1', // default
  timeout: 15_000,
});

Fifteen seconds is a sane default for chat completions; we’ll wire per-attempt AbortSignals later for stricter bounds.

Defining the fallback chain

A fallback chain is an ordered array of model identifiers. Order matters: put the best quality or cheapest model first depending on your SLA.

// models.ts
export const MODEL_CHAIN = [
  'gpt-4o',
  'gpt-4o-mini',
  'gpt-3.5-turbo',
] as const;

export type FallbackModel = typeof MODEL_CHAIN[number];

If you use a gateway that aggregates multiple providers, the chain can mix vendors: ['anthropic/claude-3.5-sonnet', 'openai/gpt-4o']. The SDK doesn’t care as long as the endpoint routes the name.

Implementing the node.js openai sdk multi-model fallback caller

The core loop catches retryable HTTP statuses and moves to the next model. Non-retryable errors (bad request, auth failure, content filter) bubble up immediately.

// fallback.ts
import { client } from './client';
import { MODEL_CHAIN } from './models';
import type OpenAI from 'openai';

export interface FallbackResult {
  model: string;
  content: string;
  usage: OpenAI.Chat.Completions.ChatCompletion['usage'];
}

export async function chatWithFallback(
  messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[],
  models: readonly string[] = MODEL_CHAIN,
  opts: { maxTokens?: number; signal?: AbortSignal } = {},
): Promise<FallbackResult> {
  let lastErr: unknown;
  for (const model of models) {
    try {
      const resp = await client.chat.completions.create(
        {
          model,
          messages,
          max_tokens: opts.maxTokens ?? 500,
        },
        { signal: opts.signal },
      );
      const content = resp.choices[0]?.message?.content ?? '';
      return { model, content, usage: resp.usage };
    } catch (err) {
      if (err instanceof OpenAI.APIError) {
        // Retryable: rate limit (429) or any 5xx
        if (err.status === 429 || err.status >= 500) {
          lastErr = err;
          console.warn(`[fallback] ${model} returned ${err.status}; trying next`);
          continue;
        }
      }
      // Unknown error or non-retryable (401, 400, 403)
      throw err;
    }
  }
  throw new Error(
    `All ${models.length} models failed. Last error: ${String(lastErr)}`,
  );
}

Key point: we do not retry the same model in a loop. Fallback is about breadth across models, not depth on one. If you need retries, wrap this function in a separate retry decorator with exponential backoff.

Error classification details

OpenAI’s SDK throws OpenAI.APIError with a status field. In practice:

• 429 – rate limit or quota. Almost always transient.
• 500/502/503 – provider-side degradation.
• 400 – malformed request; retrying on another model won’t help.
• 401/403 – credential or entitlement issue; fail fast.

If your gateway returns 200 with an error payload (some non-OpenAI compat layers do), inspect resp.error instead. The code above assumes spec-compliant HTTP status codes.

Running the example

Wire a small entrypoint:

// index.ts
import { chatWithFallback } from './fallback';

async function main() {
  const result = await chatWithFallback([
    { role: 'user', content: 'What is the capital of France? Answer in one word.' },
  ]);
  console.log(`Model used: ${result.model}`);
  console.log(`Response: ${result.content}`);
  console.log(`Tokens: ${JSON.stringify(result.usage)}`);
}

main().catch((e) => {
  console.error('Fatal:', e);
  process.exit(1);
});

Run with npx tsx index.ts.

Expected output on success

Model used: gpt-4o
Response: Paris.
Tokens: {"prompt_tokens":14,"completion_tokens":2,"total_tokens":16}

Expected output on fallback

To see the chain engage without waiting for a real rate limit, point the first model at a mock that returns 429. The log will look like:

[fallback] gpt-4o returned 429; trying next
Model used: gpt-4o-mini
Response: Paris.
Tokens: {"prompt_tokens":14,"completion_tokens":2,"total_tokens":16}

If every model fails, you get:

Fatal: Error: All 3 models failed. Last error: OpenAI.APIError: 429 Too Many Requests

Forcing a fallback in local tests

Spin up a minimal mock server that rate-limits the first model and answers the second:

// mock.ts
import http from 'node:http';
const server = http.createServer((req, res) => {
  if (req.url?.includes('gpt-4o')) {
    res.writeHead(429);
    res.end('rate limited');
  } else {
    res.writeHead(200, { 'content-type': 'application/json' });
    res.end(JSON.stringify({
      choices: [{ message: { content: 'Paris.' } }],
      usage: { prompt_tokens: 14, completion_tokens: 2, total_tokens: 16 },
    }));
  }
});
server.listen(8787, () => console.log('mock on 8787'));

Set baseURL: 'http://localhost:8787/v1' in client.ts, run npx tsx mock.ts in one terminal and npx tsx index.ts in another. You’ll observe the exact fallback path deterministically.

Streaming considerations

Fallback with streaming is trickier: you can’t undo bytes already sent to the client. Two patterns work:

1. Buffer then stream – Run the fallback loop with stream: false to pick a model, then re-issue the request with stream: true on the chosen model. Doubles latency on the happy path but is simple and safe.
2. Stream with abort – Start streaming on model A. On first error chunk, abort the response and fall back to model B, but only if you haven’t emitted anything yet. If you have, you must fail the whole request.

For most apps, pattern 1 is simpler:

const { model } = await chatWithFallback(messages, MODEL_CHAIN, { maxTokens: 1 });
const stream = await client.chat.completions.create({
  model,
  messages,
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? '');
}

Production hardening

The naive loop is fine for a script, not for production. Add:

• Per-model timeout: pass a fresh AbortSignal with AbortSignal.timeout(8000) per iteration so a hung model doesn’t block the chain.
• Usage metering: log result.usage per model to track cost. If you use a gateway, per-token metering is often handled upstream.
• Cache-Control: forward cache_control hints if your provider supports prompt caching; some gateways honor client routing directives and forward those headers automatically.
• Structured logging: emit model, latency_ms, status for each attempt.

Example with timeout per attempt:

for (const model of models) {
  try {
    const resp = await client.chat.completions.create(
      { model, messages, max_tokens: opts.maxTokens ?? 500 },
      { signal: AbortSignal.timeout(8000) },
    );
    // ... handle resp
  } catch (err) {
    if (err instanceof DOMException && err.name === 'TimeoutError') {
      console.warn(`[fallback] ${model} timed out; trying next`);
      continue;
    }
    // ... existing APIError handling
  }
}

Also consider capping the chain length and falling back to a cached or static response if all models fail during a sustained incident.

Using a gateway with built-in fallback

Hand-rolling fallback is educational, but operating a ranked chain across providers adds overhead: key management, format drift, and status-code quirks. An OpenAI-compatible gateway such as n4n.ai exposes one endpoint across 240+ models and performs automatic fallback when a provider is rate-limited or degraded, while still honoring per-token usage metering and client routing directives. The loop you wrote above is essentially what runs server-side—knowing it helps you debug routing behavior.

Wrapping up

You now have a working node.js openai sdk multi-model fallback implementation that degrades gracefully across models. It classifies errors correctly, supports abort timeouts, and can be adapted for streaming with a buffer-first approach. Drop chatWithFallback into your service and adjust MODEL_CHAIN to match your quality/cost tradeoffs.