The Vercel AI SDK makes it straightforward to stream responses from a single model, but production systems need resilience when providers hit rate limits or degrade. This guide walks through building a robust fallback chain that automatically retries across multiple models when you encounter 429 errors, preserving streaming behavior and conversation context throughout.
Step 1: Understand the error surface
Before writing fallback logic, you need to know what the SDK actually throws. The AI SDK wraps provider errors in APICallError (from ai), which exposes statusCode, responseHeaders, and the raw response body. Rate limits typically surface as HTTP 429 with a retry-after header, but some providers return 400 with an error code like rate_limit_exceeded.
import { APICallError } from 'ai';
try {
const result = await streamText({
model: openai('gpt-4o'),
messages,
});
} catch (error) {
if (error instanceof APICallError) {
console.log('Status:', error.statusCode);
console.log('Headers:', error.responseHeaders);
console.log('Body:', error.responseBody);
}
}Run this once against a rate-limited endpoint to confirm the exact shape. Providers differ — Anthropic returns retry-after in seconds, OpenAI uses milliseconds in x-ratelimit-reset-requests, and some gateways add their own headers.
Step 2: Build a model registry with metadata
Hardcoding model strings across your codebase makes fallback brittle. Define a registry that captures each model’s provider, context window, and relative cost so your fallback policy can make intelligent decisions.
// lib/models.ts
export interface ModelSpec {
id: string;
provider: 'openai' | 'anthropic' | 'google' | 'groq' | 'custom';
contextWindow: number;
costPer1kTokens: { input: number; output: number };
supportsStreaming: boolean;
}
export const MODEL_REGISTRY: Record<string, ModelSpec> = {
'gpt-4o': {
id: 'gpt-4o',
provider: 'openai',
contextWindow: 128_000,
costPer1kTokens: { input: 0.0025, output: 0.01 },
supportsStreaming: true,
},
'gpt-4o-mini': {
id: 'gpt-4o-mini',
provider: 'openai',
contextWindow: 128_000,
costPer1kTokens: { input: 0.00015, output: 0.0006 },
supportsStreaming: true,
},
'claude-3-5-sonnet-20241022': {
id: 'claude-3-5-sonnet-20241022',
provider: 'anthropic',
contextWindow: 200_000,
costPer1kTokens: { input: 0.003, output: 0.015 },
supportsStreaming: true,
},
'llama-3.1-70b-versatile': {
id: 'llama-3.1-70b-versatile',
provider: 'groq',
contextWindow: 128_000,
costPer1kTokens: { input: 0.00059, output: 0.00079 },
supportsStreaming: true,
},
};
export type ModelId = keyof typeof MODEL_REGISTRY;This registry lets you sort fallback candidates by cost, latency, or capability without scattering magic strings.
Step 3: Create a provider factory
The AI SDK expects model instances from provider functions (openai(), anthropic(), etc.). Wrap these in a factory that reads from your registry and handles provider-specific configuration like base URLs or custom headers.
// lib/provider-factory.ts
import { openai, OpenAIProvider } from '@ai-sdk/openai';
import { anthropic, AnthropicProvider } from '@ai-sdk/anthropic';
import { groq } from '@ai-sdk/groq';
import { MODEL_REGISTRY, ModelId } from './models';
type ProviderInstance = ReturnType<typeof openai> | ReturnType<typeof anthropic> | ReturnType<typeof groq>;
export function createModel(modelId: ModelId): ProviderInstance {
const spec = MODEL_REGISTRY[modelId];
if (!spec) throw new Error(`Unknown model: ${modelId}`);
switch (spec.provider) {
case 'openai':
return openai(modelId, {
baseURL: process.env.OPENAI_BASE_URL,
headers: { 'Authorization': `Bearer ${process.env.OPENAI_API_KEY}` },
});
case 'anthropic':
return anthropic(modelId, {
baseURL: process.env.ANTHROPIC_BASE_URL,
headers: { 'x-api-key': process.env.ANTHROPIC_API_KEY },
});
case 'groq':
return groq(modelId, {
baseURL: process.env.GROQ_BASE_URL,
headers: { 'Authorization': `Bearer ${process.env.GROQ_API_KEY}` },
});
default:
throw new Error(`Unsupported provider: ${spec.provider}`);
}
}If you route through a gateway like n4n.ai, the factory becomes simpler — one base URL, one key, and the model ID passes through directly.
Step 4: Implement the fallback policy
The core logic: attempt the primary model, catch rate-limit errors, wait if retry-after is short, then escalate to the next model in the chain. Preserve the message history and streaming contract.
// lib/fallback.ts
import { streamText, StreamTextResult, APICallError, Message } from 'ai';
import { createModel } from './provider-factory';
import { MODEL_REGISTRY, ModelId } from './models';
export interface FallbackConfig {
primaryModel: ModelId;
fallbackChain: ModelId[];
maxRetriesPerModel: number;
maxTotalLatencyMs: number;
}
const DEFAULT_CONFIG: FallbackConfig = {
primaryModel: 'gpt-4o',
fallbackChain: ['gpt-4o-mini', 'claude-3-5-sonnet-20241022', 'llama-3.1-70b-versatile'],
maxRetriesPerModel: 1,
maxTotalLatencyMs: 30_000,
};
export async function streamWithFallback(
messages: Message[],
config: Partial<FallbackConfig> = {}
): Promise<StreamTextResult> {
const { primaryModel, fallbackChain, maxRetriesPerModel, maxTotalLatencyMs } = {
...DEFAULT_CONFIG,
...config,
};
const candidates = [primaryModel, ...fallbackChain];
const startTime = Date.now();
let lastError: Error | null = null;
for (const modelId of candidates) {
if (Date.now() - startTime > maxTotalLatencyMs) {
throw new Error(`Fallback chain exceeded max latency (${maxTotalLatencyMs}ms)`);
}
const model = createModel(modelId);
const spec = MODEL_REGISTRY[modelId];
for (let attempt = 0; attempt <= maxRetriesPerModel; attempt++) {
try {
console.log(`[fallback] Attempting ${modelId} (attempt ${attempt + 1})`);
return await streamText({
model,
messages,
// Truncate if the fallback model has a smaller context window
maxTokens: Math.min(spec.contextWindow, 4096),
});
} catch (error) {
lastError = error as Error;
if (!isRateLimitError(error)) {
console.log(`[fallback] Non-retryable error on ${modelId}:`, error);
break; // Move to next model immediately
}
const retryAfter = extractRetryAfter(error);
if (retryAfter && retryAfter < 5000 && attempt < maxRetriesPerModel) {
console.log(`[fallback] Rate limited, waiting ${retryAfter}ms before retry`);
await sleep(retryAfter);
continue;
}
console.log(`[fallback] Rate limit on ${modelId}, moving to next model`);
break; // Escalate to next model
}
}
}
throw new Error(`All models exhausted. Last error: ${lastError?.message}`);
}
function isRateLimitError(error: unknown): boolean {
if (error instanceof APICallError) {
if (error.statusCode === 429) return true;
// Some providers use 400 with error codes
const body = error.responseBody as Record<string, unknown> | undefined;
if (body?.error?.code === 'rate_limit_exceeded') return true;
if (body?.error?.type === 'rate_limit_error') return true;
}
return false;
}
function extractRetryAfter(error: unknown): number | null {
if (error instanceof APICallError) {
const headers = error.responseHeaders;
// Standard header (seconds)
const retryAfter = headers.get('retry-after');
if (retryAfter) return parseInt(retryAfter, 10) * 1000;
// OpenAI-specific (milliseconds)
const resetRequests = headers.get('x-ratelimit-reset-requests');
if (resetRequests) return parseInt(resetRequests, 10);
// Anthropic-specific (seconds with decimal)
const retryAfterMs = headers.get('retry-after-ms');
if (retryAfterMs) return parseInt(retryAfterMs, 10);
}
return null;
}
function sleep(ms: number): Promise<void> {
return new Promise(resolve => setTimeout(resolve, ms));
}Key design decisions: the function returns the same StreamTextResult type regardless of which model succeeds, so callers don’t need to change. Context window truncation prevents silent failures when falling back to a model with a smaller window. The maxTotalLatencyMs bound prevents the chain from running indefinitely.
Step 5: Wire it into your route handler
In a Next.js App Router route, the streaming response integrates directly with the SDK’s toDataStreamResponse() helper.
// app/api/chat/route.ts
import { streamWithFallback } from '@/lib/fallback';
import { convertToCoreMessages } from 'ai';
export async function POST(req: Request) {
const { messages } = await req.json();
const coreMessages = convertToCoreMessages(messages);
try {
const result = await streamWithFallback(coreMessages, {
primaryModel: 'gpt-4o',
fallbackChain: ['gpt-4o-mini', 'claude-3-5-sonnet-20241022'],
maxTotalLatencyMs: 25_000,
});
return result.toDataStreamResponse({
// Forward provider cache-control hints if your gateway sends them
headers: {
'Cache-Control': 'no-store, must-revalidate',
},
});
} catch (error) {
console.error('[chat] Fallback chain failed:', error);
return new Response(
JSON.stringify({ error: 'All models unavailable, please try again' }),
{ status: 503, headers: { 'Content-Type': 'application/json' } }
);
}
}The convertToCoreMessages call normalizes the client’s message format (which may include data parts for tool calls) into the SDK’s internal representation.
Step 6: Add observability
You can’t debug fallback behavior in production without structured logs. Emit events at each decision point — model selected, attempt started, rate limit hit, retry wait, escalation, success.
// lib/telemetry.ts
export interface FallbackEvent {
timestamp: string;
stage: 'attempt' | 'retry' | 'escalate' | 'success' | 'exhausted';
model: string;
attempt: number;
latencyMs?: number;
errorCode?: string;
retryAfterMs?: number;
}
const listeners: Array<(event: FallbackEvent) => void> = [];
export function onFallbackEvent(listener: (event: FallbackEvent) => void) {
listeners.push(listener);
}
export function emitFallbackEvent(event: FallbackEvent) {
const enriched = { ...event, timestamp: new Date().toISOString() };
console.log('[fallback]', JSON.stringify(enriched));
listeners.forEach(fn => fn(enriched));
}Then sprinkle emitFallbackEvent calls throughout streamWithFallback. In development, the console output is enough. In production, pipe to your logging pipeline (Datadog, Axiom, etc.) and build a dashboard showing fallback frequency by model, average retry latency, and chain exhaustion rate.
Step 7: Handle conversation context across fallbacks
When you switch models mid-conversation, the new model sees the full history including the previous model’s partial response (if any). This is usually fine, but two edge cases matter:
-
Tool calls: If the primary model emitted a tool call that hasn’t returned yet, the fallback model will see an incomplete tool call sequence. Either abort the tool call before falling back, or design your tool schema to be idempotent.
-
System prompt drift: Different models interpret system prompts differently. Keep your system prompt in the message array (not baked into the model call) so it travels with the conversation.
// Ensure system prompt is always first
const messagesWithSystem: Message[] = [
{ role: 'system', content: SYSTEM_PROMPT },
...coreMessages.filter(m => m.role !== 'system'),
];Step 8: Verify the fallback chain works
Write an integration test that forces a rate limit on the primary model and asserts the fallback succeeds. Use a mock provider or a test endpoint that returns 429.
// __tests__/fallback.test.ts
import { streamWithFallback } from '@/lib/fallback';
import { MockModelProvider } from 'ai/test';
describe('streamWithFallback', () => {
it('falls back to second model on rate limit', async () => {
const failingProvider = new MockModelProvider({
responses: [{ status: 429, headers: { 'retry-after': '1' } }],
});
const succeedingProvider = new MockModelProvider({
responses: [{ content: 'fallback response' }],
});
// Patch the factory for this test
jest.spyOn(require('@/lib/provider-factory'), 'createModel')
.mockImplementation((id: string) => {
if (id === 'gpt-4o') return failingProvider;
if (id === 'gpt-4o-mini') return succeedingProvider;
throw new Error(`Unexpected model: ${id}`);
});
const result = await streamWithFallback([
{ role: 'user', content: 'Hello' },
], {
primaryModel: 'gpt-4o',
fallbackChain: ['gpt-4o-mini'],
maxRetriesPerModel: 0,
});
const chunks = [];
for await (const chunk of result.textStream) {
chunks.push(chunk);
}
expect(chunks.join('')).toBe('fallback response');
});
});Run this in CI. It catches regressions when provider error formats change or when the registry gets out of sync with the factory.
Step 9: Manual verification checklist
Before deploying, exercise the full chain manually:
- Primary succeeds: Send a normal request, verify streaming works end-to-end.
- Primary rate-limits: Temporarily set your primary model’s API key to an invalid value that returns 429 (or use a test endpoint). Confirm the fallback model responds.
- Chain exhaustion: Invalidate all keys. Verify the 503 response with a clean error message.
- Context preservation: Start a multi-turn conversation, force a fallback on turn 2, verify the model references turn 1 correctly.
- Latency budget: Add artificial delay to the primary model, confirm
maxTotalLatencyMscuts off the chain.
Step 10: Tune the policy for your workload
The default chain prioritizes capability similarity (GPT-4o → GPT-4o-mini → Claude Sonnet). Adjust based on your constraints:
- Cost-sensitive: Put cheaper models earlier in the chain, accept quality degradation.
- Latency-sensitive: Put fastest providers (Groq, Together) first, accept smaller context windows.
- Capability-sensitive: Keep same-provider fallbacks together (OpenAI → OpenAI) to preserve tool-calling behavior and output format.
Track your fallback rate weekly. If you’re falling back >5% of requests, the primary model’s quota is the bottleneck — request a limit increase or shard traffic across multiple API keys.
The fallback chain is now a first-class citizen in your architecture, not an afterthought. The same pattern extends to other failure modes: context window overflow (fallback to larger-window model), capability mismatch (fallback to model that supports tools), or provider degradation (fallback to different provider entirely). The registry + factory + policy structure keeps each concern isolated and testable.