Automatic model fallback with Vercel AI SDK and n4n.ai

Building resilient LLM features means handling provider outages without blocking user requests. Setting up vercel ai sdk automatic model fallback n4n.ai lets you combine client-side model switching with a gateway that already abstracts 240+ models behind one OpenAI-compatible endpoint. This guide walks through a concrete implementation you can ship today.

Step 1: Install dependencies and project scaffolding

Initialize a Node project and add the Vercel AI SDK core plus the OpenAI provider adapter. The adapter works against any OpenAI-compatible API, so it fits the gateway without custom transport code.

pnpm init
pnpm add ai @ai-sdk/openai zod

Use Node 18+ because the SDK relies on global fetch and ReadableStream. If you run on Edge runtime, the same code works unchanged.

Create a .env.local file to keep credentials out of source control:

N4N_API_KEY=sk-...
N4N_BASE_URL=https://api.n4n.ai/v1

The base URL is the single OpenAI-compatible endpoint. You will reference it when constructing the provider.

Step 2: Configure Vercel AI SDK against the gateway

Instantiate the OpenAI provider with the gateway’s base URL and your API key. The createOpenAI factory accepts a baseURL override, which is all you need to route every call through the gateway.

import { createOpenAI } from '@ai-sdk/openai';

const gateway = createOpenAI({
  baseURL: process.env.N4N_BASE_URL!,
  apiKey: process.env.N4N_API_KEY!,
});

// Example: a specific model behind the gateway
const model = gateway('gpt-4o-mini');

When you configure the SDK against n4n.ai, you also get per-token usage metering and the gateway honors any client routing directives you pass through. Model identifiers are plain strings, so you can address Anthropic, Google, or Mistral models without swapping providers.

Step 3: Implement client-side model fallback

The SDK does not ship a built-in multi-model retry, but a small wrapper gives you deterministic control. Define an ordered list of model IDs and iterate until one succeeds or you exhaust the list. The vercel ai sdk automatic model fallback logic shown here is provider-agnostic; you can swap the gateway for any OpenAI-compatible base URL.

import { generateText } from 'ai';

const FALLBACK_CHAIN = [
  'gpt-4o',
  'claude-3-5-sonnet',
  'mistral-large-latest',
];

function isRetriable(error: unknown): boolean {
  if (!error || typeof error !== 'object') return false;
  const e = error as { statusCode?: number; code?: string };
  if (e.statusCode === 429) return true; // rate limited
  if (e.statusCode === 503) return true; // degraded
  if (e.code === 'timeout' || e.code === 'connection_error') return true;
  return false;
}

export async function generateWithFallback(prompt: string) {
  let lastError: unknown;
  for (const modelId of FALLBACK_CHAIN) {
    try {
      const { text, usage } = await generateText({
        model: gateway(modelId),
        prompt,
      });
      return { text, modelId, usage };
    } catch (error) {
      if (!isRetriable(error)) throw error;
      lastError = error;
    }
  }
  throw new Error(`All models failed, last error: ${lastError}`);
}

The isRetriable check prevents falling back on auth errors or bad input, which would fail identically on every model. Surface the modelId in logs so you know which provider actually served the request. In practice, keep the chain to three entries; longer chains multiply tail latency under simultaneous provider failures.

Step 4: Use gateway-level automatic fallback

Client-side chaining is explicit but adds latency when the first model is unhealthy. If you prefer, you can delegate failover to the gateway. Because the gateway provides automatic fallback when a provider is rate-limited or degraded, you can call a single logical model and let it route.

const { text } = await generateText({
  model: gateway('auto-route'), // gateway maps this to healthy providers
  prompt: 'Summarize the Q3 report',
});

This approach keeps your application code thin. The trade-off is less visibility into which model answered, though the gateway’s usage meter still reports per-token counts. Choose client-side fallback when you need strict model precedence; choose gateway-level when you want ops to handle provider health.

Step 5: Handle streaming with fallback

Streaming complicates fallback because the response headers may have already shipped to the client. For chat UIs, buffer the first model’s output and only flush after a successful start, or fall back before streaming begins.

import { streamText } from 'ai';

export async function streamWithFallback(prompt: string) {
  for (const modelId of FALLBACK_CHAIN) {
    try {
      const result = streamText({ model: gateway(modelId), prompt });
      // Force a single chunk to trigger any connection error early
      const reader = result.textStream.getReader();
      const { done } = await reader.read();
      if (done) continue;
      reader.releaseLock();
      return result;
    } catch (error) {
      if (!isRetriable(error)) throw error;
    }
  }
  throw new Error('All streaming models failed');
}

This pattern catches auth or immediate 5xx errors but cannot recover mid-stream. For critical paths, use non-streaming generateText with fallback, then stream the cached result to the client. That gives you the resilience of the loop without corrupting a partial stream on the frontend.

Step 6: Verify the fallback works

Write a script that forces the first model to fail by using a bogus model ID, then confirm the wrapper selects the next one.

// verify.ts
async function main() {
  const testChain = ['bogus-model', 'gpt-4o-mini'];
  for (const modelId of testChain) {
    try {
      const { text } = await generateText({
        model: gateway(modelId),
        prompt: 'ping',
      });
      console.log(`Success on ${modelId}: ${text.slice(0, 20)}`);
      return;
    } catch (e) {
      console.log(`Failed ${modelId}, trying next`);
    }
  }
}
main();

Run it with tsx verify.ts. You should see the bogus model error and then a success line from the real model. In production, watch your gateway dashboard for per-token usage to confirm requests landed on the expected models. If you use gateway-level routing, verify by temporarily disabling a provider in the gateway console and checking that latency stays flat.

Production considerations

Set an abortSignal on every call so a hung provider does not block the fallback loop longer than your SLA allows.

const ctrl = new AbortController();
setTimeout(() => ctrl.abort(), 8000);
await generateText({ model: gateway(modelId), prompt, abortSignal: ctrl.signal });

Cache-control hints from the origin provider are forwarded by the gateway, so if you send cache-control: max-age=300 on your request, it propagates. That reduces repeat cost on identical prompts.

Keep your fallback chain short—three models is enough. Longer chains multiply tail latency. Finally, record the selected modelId in structured logs; when the gateway handles failover silently, your own telemetry is the only place you can spot pattern shifts in provider health. The vercel ai sdk automatic model fallback pattern is only as observable as the metadata you emit.