n4nAI
Open console →
API integrationTutorialMulti-Provider Fallback Code Patterns

Multi-provider LLM fallback code patterns for Node.js

Build a multi provider llm fallback nodejs tutorial: sequential and parallel patterns, timeouts, circuit breakers, and cache hints.

n4n Team3 min read733 words

Audio narration

Coming soon — every post will get a voice note here.

When you wire LLM calls into production, provider outages and rate limits are not edge cases. A solid multi provider llm fallback nodejs pattern keeps your app responsive when OpenAI 429s or a smaller provider is degraded. This tutorial builds a small but real fallback client from scratch using nothing but Node’s built-in fetch and a few dozen lines of TypeScript.

Prerequisites

  • Node.js 18.0.0 or later (global fetch, AbortController, and Promise.any are available)
  • API keys for at least two OpenAI-compatible endpoints (OpenAI, Groq, Together, or a gateway)
  • A scratch directory: npm init -y is enough — we add zero runtime dependencies
  • Basic comfort with async/await and HTTP status codes

If you only have one key, point both provider entries at the same host to exercise the success path, then temporarily typo the URL to simulate a failure.

The baseline single call

All providers we target expose /v1/chat/completions with the same JSON shape. Start there so the fallback logic has a clean primitive to wrap.

async function chatOnce(baseUrl: string, apiKey: string, model: string, prompt: string) {
  const resp = await fetch(`${baseUrl}/v1/chat/completions`, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      authorization: `Bearer ${apiKey}`,
    },
    body: JSON.stringify({
      model,
      messages: [{ role: "user", content: prompt }],
      max_tokens: 128,
    }),
  });
  if (!resp.ok) {
    throw new Error(`HTTP ${resp.status}: ${await resp.text()}`);
  }
  const data = await resp.json();
  return data.choices[0].message.content as string;
}

Run it once to confirm wiring:

node --input-type=module -e '
import { chatOnce } from "./client.mjs";
chatOnce("https://api.openai.com", process.env.OPENAI_KEY, "gpt-4o-mini", "say hi")
  .then(console.log).catch(console.error);
'

Expected output is a short greeting string. If you see HTTP 401, your key is wrong; HTTP 429 means you’ve hit a rate limit and are already in fallback territory.

Sequential fallback

The simplest multi provider llm fallback nodejs approach is an ordered try/catch loop. Define a provider list and walk it until one returns.

const PROVIDERS = [
  { baseUrl: "https://api.openai.com", apiKey: process.env.OPENAI_KEY, model: "gpt-4o-mini" },
  { baseUrl: "https://api.groq.com", apiKey: process.env.GROQ_KEY, model: "llama3-8b-8192" },
];

async function chatFallbackSequential(prompt: string) {
  let lastErr: unknown;
  for (const p of PROVIDERS) {
    try {
      const text = await chatOnce(p.baseUrl, p.apiKey, p.model, prompt);
      console.log(`[ok] ${p.baseUrl}`);
      return text;
    } catch (err) {
      console.warn(`[fail] ${p.baseUrl}: ${err.message}`);
      lastErr = err;
    }
  }
  throw new Error(`All providers failed: ${lastErr}`);
}

Checkpoint output when the first provider is healthy:

[ok] https://api.openai.com
Hello! How can I help you?

If OpenAI is down, you’ll see [fail] https://api.openai.com: HTTP 503 followed by [ok] https://api.groq.com. That loop is the core of the pattern.

Error taxonomy before you add complexity

Not every error should trigger a fallback. A 400 from malformed input will fail identically on the next provider. A 429 or 5xx is a transient infrastructure signal. Filter inside the catch:

function isTransient(err: any) {
  const status = err?.message?.match(/HTTP (\d+)/)?.[1];
  return status === "429" || (status && status.startsWith("5"));
}

Use it in the loop: only continue to the next provider when isTransient(err) is true; otherwise rethrow immediately. This avoids burning quota on guaranteed failures.

Timeouts and abort

A hung connection is worse than a fast 500. Wrap each call in an AbortController with a budget so a slow provider is skipped instead of blocking the user request.

async function chatOnceTimeout(
  baseUrl: string, apiKey: string, model: string, prompt: string, ms = 8000
) {
  const ctrl = new AbortController();
  const t = setTimeout(() => ctrl.abort(), ms);
  try {
    const resp = await fetch(`${baseUrl}/v1/chat/completions`, {
      method: "POST",
      signal: ctrl.signal,
      headers: { "content-type": "application/json", authorization: `Bearer ${apiKey}` },
      body: JSON.stringify({ model, messages: [{ role: "user", content: prompt }], max_tokens: 128 }),
    });
    if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
    const data = await resp.json();
    return data.choices[0].message.content as string;
  } finally {
    clearTimeout(t);
  }
}

Plug chatOnceTimeout into the loop and a provider that sits on the socket gets aborted at 8 seconds. The next entry tries immediately.

Circuit breaker to avoid wasted calls

If a provider has been failing repeatedly, don’t even send the request. A tiny in-memory state map suffices for a single process.

const breaker: Record<string, { fails: number; until: number }> = {};

function shouldSkip(baseUrl: string) {
  const b = breaker[baseUrl];
  return b !== undefined && b.until > Date.now();
}

function recordFail(baseUrl: string) {
  const b = breaker[baseUrl] ?? { fails: 0, until: 0 };
  b.fails++;
  if (b.fails >= 3) b.until = Date.now() + 10_000;
  breaker[baseUrl] = b;
}

function recordSuccess(baseUrl: string) {
  breaker[baseUrl] = { fails: 0, until: 0 };
}

Integrate inside the loop: skip when shouldSkip(p.baseUrl), call recordFail on transient error, and recordSuccess on return. This keeps the multi provider llm fallback nodejs client from hammering a dead endpoint and wasting latency budget.

Parallel race pattern

Sequential fallback adds latency equal to the sum of timeouts when the first N providers are dead. If you can tolerate potential double-billing, fire all requests and take the first settled.

async function chatRace(prompt: string, timeoutMs = 8000) {
  const controllers = PROVIDERS.map(() => new AbortController());
  const tasks = PROVIDERS.map((p, i) => {
    const ctrl = controllers[i];
    const t = setTimeout(() => ctrl.abort(), timeoutMs);
    return chatOnceTimeout(p.baseUrl, p.apiKey, p.model, prompt, timeoutMs)
      .finally(() => clearTimeout(t))
      .then((text) => {
        controllers.forEach((c, j) => j !== i && c.abort());
        return { baseUrl: p.baseUrl, text };
      });
  });
  const winner = await Promise.any(tasks);
  console.log(`[race winner] ${winner.baseUrl}`);
  return winner.text;
}

Promise.any rejects only if every task rejects, which matches fallback semantics. The first resolve aborts its siblings so you don’t keep streaming tokens from losers.

Honoring cache-control and routing hints

Some providers support prompt caching via request body fields or headers. When you forward a user request through your own stack, pass those through instead of dropping them. In self-built code you map per provider:

function buildBody(model: string, prompt: string, cacheHint?: boolean) {
  const body: any = { model, messages: [{ role: "user", content: prompt }], max_tokens: 128 };
  if (cacheHint) body.cache_control = { type: "ephemeral" }; // Anthropic-style, ignored by others
  return body;
}

If a client sends a routing directive like x-route-to: groq, copy only allow-listed headers before calling fetch. A unified OpenAI-compatible gateway typically forwards provider cache-control hints and routing automatically; in your own loop you own that mapping.

A cohesive FallbackClient

class FallbackClient {
  constructor(private providers: typeof PROVIDERS) {}

  async chat(prompt: string, opts: { timeoutMs?: number; cache?: boolean } = {}) {
    for (const p of this.providers) {
      if (shouldSkip(p.baseUrl)) continue;
      try {
        const ctrl = new AbortController();
        const t = setTimeout(() => ctrl.abort(), opts.timeoutMs ?? 8000);
        const resp = await fetch(`${p.baseUrl}/v1/chat/completions`, {
          signal: ctrl.signal,
          method: "POST",
          headers: { "content-type": "application/json", authorization: `Bearer ${p.apiKey}` },
          body: JSON.stringify(buildBody(p.model, prompt, opts.cache)),
        });
        clearTimeout(t);
        if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
        recordSuccess(p.baseUrl);
        const data = await resp.json();
        return data.choices[0].message.content as string;
      } catch (err) {
        if (isTransient(err)) recordFail(p.baseUrl);
        else throw err;
      }
    }
    throw new Error("all providers failed");
  }
}

Instantiate with your list and call .chat("summarize this log"). The class encapsulates the multi provider llm fallback nodejs logic so the rest of your app just sees a string.

Testing the fallback locally

Force a failure without touching real keys. Spin up a one-line dead server:

node --input-type=module -e '
import { createServer } from "http";
createServer((_, res) => res.writeHead(500).end("down")).listen(9999);
'

Point the first provider at http://127.0.0.1:9999 and run chatFallbackSequential. You should see:

[fail] http://127.0.0.1:9999: HTTP 500
[ok] https://api.groq.com
(actual model output)

That confirms the fallback fired before any real user-facing timeout.

When managed fallback makes sense

Hand-rolling timeouts, breakers, and cache mapping is fine until you support 20 providers and need per-token accounting. An OpenAI-compatible gateway such as n4n.ai offers automatic fallback when a provider is rate-limited or degraded, per-token usage metering, and honors client routing directives—offloading the exact logic above. Use the code here when you need full control of each request; use a gateway when you’d rather ship product features.

Checkpoint: full run output

[fail] https://api.openai.com: HTTP 429
[ok] https://api.groq.com
Here is a summary: the service is healthy.

That’s the entire multi provider llm fallback nodejs stack: sequential with transient-error filtering, timeout abort, circuit breaker, and a parallel race variant for low-latency needs. Swap the provider array for your own, add metrics around recordFail, and you have production-grade resilience.

Tagsfallbacknodejsmulti-providercode-pattern

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
API integrationMulti-provider LLM fallback code patterns in GoA hands-on Go tutorial for building multi provider llm fallback golang patterns: chain providers, handle rate limits, and degrade gracefully.API integrationMulti-provider LLM fallback: a Python code patternA hands-on Python tutorial for building a multi provider llm fallback python pattern with OpenAI and Anthropic SDKs, plus graceful degradation.API integrationA fallback pattern for LLM APIs using try/except chainsBuild a Python try except fallback pattern llm api tutorial: chain multiple providers with graceful degradation, retries, and clear error handling.API integrationCircuit breakers for multi-provider LLM fallback codeA practical guide to implementing a circuit breaker llm fallback pattern across multiple LLM providers, including Python code, tradeoffs, and common pitfalls.API integrationHandling HTTP 429s with multi-provider LLM fallback codeStep-by-step guide to handle 429 rate limit llm fallback across multiple LLM providers with practical Python code, backoff, circuit breakers, and verification.API integrationHow to chain three LLM providers as automatic fallbacksLearn how to chain LLM providers with automatic fallback in Python, using OpenAI-compatible endpoints and robust error handling for production.

All multi-provider fallback code patterns posts →