Building resilient AI applications means expecting providers to fail in different ways. You need typed error handling llm fallback that separates retryable outages from permanent input mistakes, otherwise your fallback chain will either mask bugs or waste latency on doomed requests.
The failure modes you actually hit
When you call an LLM API, errors arrive in three layers: transport, HTTP status, and semantic. Transport failures are network timeouts or DNS issues. HTTP statuses split into 4xx (client error) and 5xx (server error). Semantic errors are valid HTTP 200 responses that contain garbage, missing fields, or fail your schema validation.
A naive try/catch that retries on any exception will retry on a 401 and burn a second provider’s quota for nothing. Typed error handling llm fallback forces you to make that decision explicit.
Define a provider-agnostic error hierarchy
Use a discriminated union on type. This gives exhaustiveness checking in TypeScript and makes fallback logic readable.
export type LLMError =
| { type: 'network'; provider: string; cause: Error }
| { type: 'rate_limit'; provider: string; retryAfterMs?: number }
| { type: 'auth'; provider: string; status: number }
| { type: 'invalid_request'; provider: string; message: string }
| { type: 'model_unavailable'; provider: string; model: string }
| { type: 'provider_error'; provider: string; status: number; body: unknown }
| { type: 'output_validation'; provider: string; message: string };
export type ChatRequest = {
model: string;
messages: { role: 'system' | 'user' | 'assistant'; content: string }[];
temperature?: number;
};
export type ChatResponse = {
id: string;
model: string;
choices: { message: { role: 'assistant'; content: string } }[];
};Wrap each provider in a uniform interface
Define a thin client that converts native SDK or fetch errors into LLMError. Below is a minimal OpenAI-compatible implementation using fetch.
interface ChatProvider {
name: string;
complete(req: ChatRequest): Promise<ChatResponse>;
}
async function openAIComplete(
baseUrl: string,
apiKey: string,
req: ChatRequest
): Promise<ChatResponse> {
try {
const res = await fetch(`${baseUrl}/v1/chat/completions`, {
method: 'POST',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${apiKey}`,
},
body: JSON.stringify(req),
});
if (!res.ok) {
const text = await res.text();
if (res.status === 429)
throw <LLMError>{ type: 'rate_limit', provider: baseUrl, retryAfterMs: Number(res.headers.get('retry-after') ?? 1000) };
if (res.status === 401 || res.status === 403)
throw <LLMError>{ type: 'auth', provider: baseUrl, status: res.status };
if (res.status === 400)
throw <LLMError>{ type: 'invalid_request', provider: baseUrl, message: text };
if (res.status === 404)
throw <LLMError>{ type: 'model_unavailable', provider: baseUrl, model: req.model };
throw <LLMError>{ type: 'provider_error', provider: baseUrl, status: res.status, body: text };
}
const json = await res.json();
if (!json.choices?.length) {
throw <LLMError>{ type: 'output_validation', provider: baseUrl, message: 'empty choices' };
}
return json as ChatResponse;
} catch (e) {
if ((e as LLMError).type) throw e;
throw <LLMError>{ type: 'network', provider: baseUrl, cause: e as Error };
}
}The <LLMError> cast is a shorthand; in real code you’d use a factory function to avoid repetition and attach timestamps.
Implement the fallback chain
The core primitive is a loop over providers with a retryability predicate. Only errors that pass the predicate trigger the next provider.
function isRetryable(e: LLMError): boolean {
switch (e.type) {
case 'network':
case 'rate_limit':
case 'model_unavailable':
case 'provider_error':
return true;
case 'auth':
case 'invalid_request':
case 'output_validation':
return false;
}
}
async function withFallback(
providers: ChatProvider[],
req: ChatRequest,
retryable: (e: LLMError) => boolean = isRetryable
): Promise<ChatResponse> {
let lastErr: LLMError | null = null;
for (const p of providers) {
try {
return await p.complete(req);
} catch (e) {
const err = e as LLMError;
if (!retryable(err)) throw err;
lastErr = err;
}
}
throw lastErr ?? new Error('no providers configured');
}Order matters. Put the cheapest or fastest provider first, but ensure the fallback target supports the same capabilities (tool calls, JSON mode) as the primary.
Classifying output validation separately
A 200 response that fails your Zod schema is not a provider outage. Mark it output_validation. By default, do not retry on it—the same prompt will likely fail on the next provider too. If you have a secondary prompt-repair step, handle that outside the provider chain.
import { z } from 'zod';
const ResponseSchema = z.object({
choices: z.array(z.object({ message: z.object({ content: z.string() }) })),
});
function validate(res: ChatResponse): ChatResponse {
const parsed = ResponseSchema.safeParse(res);
if (!parsed.success) {
throw <LLMError>{ type: 'output_validation', provider: 'unknown', message: parsed.error.message };
}
return res;
}Gateway vs client-side responsibility
A gateway such as n4n.ai can perform automatic fallback when a provider is rate-limited or degraded, shielding you from 429/5xx noise. But you still need typed error handling llm fallback in your own layer to catch output_validation and invalid_request errors, because those are semantic and the gateway cannot know your schema. Use the gateway for transport-level resilience; keep the typed union for application logic.
Streaming changes the calculus
With streaming, you get a 200 and then chunks. A mid-stream abort means you already consumed tokens. Implement fallback only before the first token arrives. After that, surface the error to the user and offer a manual retry.
async function* streamWithFallback(
providers: ChatProvider[],
req: ChatRequest
): AsyncGenerator<string> {
for (const p of providers) {
try {
for await (const chunk of await p.stream(req)) {
yield chunk;
}
return;
} catch (e) {
if ((e as LLMError).type !== 'network' && (e as LLMError).type !== 'provider_error') throw e;
// else try next provider, but only if no tokens yielded yet
}
}
}In practice, most teams abort the whole request on first error and restart, accepting duplicated prompt tokens on the fallback.
Testing the chain
Mock providers that throw specific LLMError subtypes. Verify non-retryable errors bubble immediately.
import { describe, it, expect } from 'vitest';
const failingAuth: ChatProvider = {
name: 'auth-fail',
complete: async () => { throw <LLMError>{ type: 'auth', provider: 'x', status: 401 }; },
};
const working: ChatProvider = {
name: 'ok',
complete: async (req) => ({ id: '1', model: req.model, choices: [{ message: { role: 'assistant', content: 'hi' } }] }),
};
describe('withFallback', () => {
it('does not fall back on auth error', async () => {
await expect(withFallback([failingAuth, working], { model: 'gpt', messages: [] }))
.rejects.toMatchObject({ type: 'auth' });
});
});Common pitfalls and tradeoffs
Double billing. If provider A streams 50 tokens then errors, and you fall back to B, you pay for the prompt twice and possibly partial generation on A. Gate fallback on pre-first-token state.
Latency tail. Sequential fallback adds the timeout of A plus B. Set aggressive AbortController timeouts (e.g., 8s) on the primary before moving on.
Capability drift. Falling back from a 70B model to a 7B model changes output quality. Encode minimum capability tags in your provider list and filter.
Lost context. Always attach provider and status to your error logs. A generic “LLM failed” alert is useless for debugging.
Over-retry of invalid requests. If your prompt exceeds context length, every provider will 400. Make invalid_request non-retryable and fix the caller.
Typed error handling llm fallback is not just defensive coding; it’s the contract that lets you swap models without rewriting business logic. Build the union once, wrap every provider the same way, and your fallback chain becomes a configuration detail rather than a tangle of catch blocks.