LangChain.js error handling retry fallback patterns are essential once you move beyond prototypes. Network partitions, rate limits, and provider outages will hit your application — often simultaneously. This guide walks through building a resilient invocation layer that retries transient failures, falls back across models and providers, and surfaces actionable diagnostics.
Step 1: categorize the failures you actually see
Before writing retry logic, instrument your calls to understand what fails. LangChain.js throws errors that wrap provider responses, but the error classes vary by integration. Start by adding a lightweight wrapper that logs error metadata without swallowing stack traces.
// lib/llm/instrumented.ts
import { BaseChatModel } from "@langchain/core/language_models/chat_models";
import { BaseMessage } from "@langchain/core/messages";
export interface LLMError extends Error {
statusCode?: number;
provider?: string;
model?: string;
retryable?: boolean;
headers?: Record<string, string>;
}
export async function invokeWithMetadata(
model: BaseChatModel,
messages: BaseMessage[],
options: { provider: string; modelName: string } = { provider: "unknown", modelName: "unknown" }
) {
try {
return await model.invoke(messages);
} catch (err: any) {
const enhanced: LLMError = new Error(err.message) as LLMError;
enhanced.statusCode = err.statusCode ?? err.response?.status;
enhanced.provider = options.provider;
enhanced.model = options.modelName;
enhanced.headers = err.response?.headers;
enhanced.retryable = isRetryable(enhanced);
throw enhanced;
}
}
function isRetryable(err: LLMError): boolean {
if (err.statusCode === 429) return true; // rate limit
if (err.statusCode === 502 || err.statusCode === 503 || err.statusCode === 504) return true; // upstream
if (err.statusCode === 408) return true; // request timeout
// Provider-specific: OpenAI returns 500 on some transient overloads
if (err.statusCode === 500 && err.provider === "openai") return true;
return false;
}Run a load test against your current models for ten minutes and bucket errors by statusCode and provider. You will typically see 429s from rate limits, 502/503/504 from gateway issues, and occasional 500s from model servers. This data drives your retry and fallback policy.
Step 2: implement exponential backoff with jitter
LangChain.js does not ship a built-in retry utility. Use async-retry or write a small helper — the latter keeps dependencies minimal and lets you inject custom logic like respecting Retry-After headers.
// lib/llm/retry.ts
import { invokeWithMetadata, LLMError } from "./instrumented";
import { BaseChatModel } from "@langchain/core/language_models/chat_models";
import { BaseMessage } from "@langchain/core/messages";
export interface RetryPolicy {
maxAttempts: number;
baseDelayMs: number;
maxDelayMs: number;
jitterFactor: number; // 0-1
}
export const DEFAULT_RETRY_POLICY: RetryPolicy = {
maxAttempts: 3,
baseDelayMs: 500,
maxDelayMs: 8000,
jitterFactor: 0.3,
};
export async function invokeWithRetry(
model: BaseChatModel,
messages: BaseMessage[],
policy: RetryPolicy = DEFAULT_RETRY_POLICY,
meta: { provider: string; modelName: string }
) {
let attempt = 0;
let lastError: LLMError;
while (attempt < policy.maxAttempts) {
try {
return await invokeWithMetadata(model, messages, meta);
} catch (err: any) {
lastError = err;
attempt++;
if (attempt >= policy.maxAttempts || !err.retryable) {
throw err;
}
const delay = calculateDelay(attempt, policy, err);
await sleep(delay);
}
}
throw lastError!;
}
function calculateDelay(attempt: number, policy: RetryPolicy, err: LLMError): number {
// Honor Retry-After if present (seconds or HTTP-date)
const retryAfter = err.headers?.["retry-after"];
if (retryAfter) {
const seconds = parseRetryAfter(retryAfter);
if (!isNaN(seconds)) return Math.min(seconds * 1000, policy.maxDelayMs);
}
const exponential = policy.baseDelayMs * Math.pow(2, attempt - 1);
const capped = Math.min(exponential, policy.maxDelayMs);
const jitter = capped * policy.jitterFactor * Math.random();
return Math.floor(capped + jitter);
}
function parseRetryAfter(value: string): number {
const asSeconds = Number(value);
if (!isNaN(asSeconds)) return asSeconds;
const date = Date.parse(value);
if (!isNaN(date)) return Math.max(0, (date - Date.now()) / 1000);
return NaN;
}
function sleep(ms: number): Promise<void> {
return new Promise(resolve => setTimeout(resolve, ms));
}Verify: Point this at a test endpoint that returns 429 on the first two calls then succeeds. Confirm the third attempt returns a valid response and total latency matches your backoff curve.
Step 3: build a fallback chain across models and providers
Retry handles transient blips. Fallback handles sustained degradation — a provider hitting quota, a model deprecated, or a region outage. Design the chain as an ordered list of (model, metadata) tuples. The first successful invocation wins.
// lib/llm/fallback.ts
import { BaseChatModel } from "@langchain/core/language_models/chat_models";
import { BaseMessage, AIMessage } from "@langchain/core/messages";
import { invokeWithRetry, DEFAULT_RETRY_POLICY } from "./retry";
export interface FallbackCandidate {
model: BaseChatModel;
provider: string;
modelName: string;
retryPolicy?: typeof DEFAULT_RETRY_POLICY;
}
export interface FallbackResult {
response: AIMessage;
used: { provider: string; modelName: string; attempt: number };
errors: Array<{ provider: string; modelName: string; error: Error }>;
}
export async function invokeWithFallback(
candidates: FallbackCandidate[],
messages: BaseMessage[]
): Promise<FallbackResult> {
const errors: FallbackResult["errors"] = [];
for (const candidate of candidates) {
const policy = candidate.retryPolicy ?? DEFAULT_RETRY_POLICY;
try {
const response = await invokeWithRetry(
candidate.model,
messages,
policy,
{ provider: candidate.provider, modelName: candidate.modelName }
);
return {
response,
used: { provider: candidate.provider, modelName: candidate.modelName, attempt: 1 },
errors,
};
} catch (err: any) {
errors.push({
provider: candidate.provider,
modelName: candidate.modelName,
error: err,
});
// Continue to next candidate
}
}
// All exhausted — throw aggregated error
const aggregate = new Error("All fallback candidates exhausted") as Error & { fallbacks: typeof errors };
aggregate.fallbacks = errors;
throw aggregate;
}Wire it up with concrete models. Order matters: put your preferred (cost/quality) model first, then cheaper fallbacks, then a last-resort model with generous limits.
// lib/llm/chain.ts
import { ChatOpenAI } from "@langchain/openai";
import { ChatAnthropic } from "@langchain/anthropic";
import { ChatGoogleGenerativeAI } from "@langchain/google-genai";
import { HumanMessage } from "@langchain/core/messages";
import { invokeWithFallback, FallbackCandidate } from "./fallback";
function buildCandidates(): FallbackCandidate[] {
return [
{
model: new ChatOpenAI({ model: "gpt-4o", temperature: 0 }),
provider: "openai",
modelName: "gpt-4o",
},
{
model: new ChatAnthropic({ model: "claude-3-5-sonnet-20240620", temperature: 0 }),
provider: "anthropic",
modelName: "claude-3-5-sonnet",
},
{
model: new ChatGoogleGenerativeAI({ model: "gemini-1.5-pro", temperature: 0 }),
provider: "google",
modelName: "gemini-1.5-pro",
retryPolicy: { ...DEFAULT_RETRY_POLICY, maxAttempts: 2 }, // faster failover
},
];
}
export async function runWithResilience(prompt: string) {
const candidates = buildCandidates();
const messages = [new HumanMessage(prompt)];
return invokeWithFallback(candidates, messages);
}Verify: Set OPENAI_API_KEY=invalid and run. The call should fail over to Anthropic, then Google, and finally throw an aggregated error with all three failures recorded. Restore the key and confirm the first candidate succeeds.
Step 4: handle provider-specific quirks
Each provider surfaces errors differently. OpenAI includes error.code and error.type in the response body. Anthropic uses error.type with values like overloaded_error. Google returns error.code as a gRPC status. Normalize these into your LLMError so retry logic stays generic.
// lib/llm/normalize.ts
import { LLMError } from "./instrumented";
export function normalizeError(err: any, provider: string): LLMError {
const normalized: LLMError = new Error(err.message) as LLMError;
normalized.provider = provider;
switch (provider) {
case "openai": {
const body = err.response?.data ?? err.response?.body ?? err;
normalized.statusCode = err.statusCode ?? err.response?.status;
normalized.headers = err.response?.headers;
// OpenAI error codes: rate_limit_exceeded, server_error, etc.
normalized.retryable = body?.error?.code === "rate_limit_exceeded"
|| body?.error?.type === "server_error"
|| (normalized.statusCode && [429, 500, 502, 503, 504].includes(normalized.statusCode));
break;
}
case "anthropic": {
const body = err.response?.data ?? err.response?.body ?? err;
normalized.statusCode = err.statusCode ?? err.response?.status;
normalized.headers = err.response?.headers;
normalized.retryable = body?.error?.type === "overloaded_error"
|| body?.error?.type === "rate_limit_error"
|| (normalized.statusCode && [429, 502, 503, 504].includes(normalized.statusCode));
break;
}
case "google": {
normalized.statusCode = err.response?.status ?? err.code; // gRPC code
normalized.headers = err.response?.headers;
// gRPC 8 = RESOURCE_EXHAUSTED (rate limit), 14 = UNAVAILABLE
normalized.retryable = [8, 14].includes(normalized.statusCode)
|| (normalized.statusCode && [429, 502, 503, 504].includes(normalized.statusCode));
break;
}
default:
normalized.retryable = normalized.statusCode
? [429, 500, 502, 503, 504].includes(normalized.statusCode)
: false;
}
return normalized;
}Update invokeWithMetadata to call normalizeError before throwing. This keeps your retry policy in one place while respecting each provider’s semantics.
Step 5: add circuit breaking for sustained outages
If a provider returns 5xx for five consecutive requests, stop sending traffic for a cooldown period. This prevents hammering a downed service and speeds failover. A minimal circuit breaker fits in ~50 lines.
// lib/llm/circuit.ts
export enum CircuitState { Closed, Open, HalfOpen }
export interface CircuitBreakerOptions {
failureThreshold: number; // consecutive failures to open
successThreshold: number; // consecutive successes in half-open to close
cooldownMs: number; // time before half-open
}
export class CircuitBreaker {
private state = CircuitState.Closed;
private failures = 0;
private successes = 0;
private lastFailureTime = 0;
private readonly options: CircuitBreakerOptions;
constructor(options: Partial<CircuitBreakerOptions> = {}) {
this.options = {
failureThreshold: 5,
successThreshold: 2,
cooldownMs: 30_000,
...options,
};
}
async execute<T>(fn: () => Promise<T>): Promise<T> {
if (this.state === CircuitState.Open) {
if (Date.now() - this.lastFailureTime >= this.options.cooldownMs) {
this.state = CircuitState.HalfOpen;
this.successes = 0;
} else {
throw new Error("Circuit open");
}
}
try {
const result = await fn();
this.onSuccess();
return result;
} catch (err) {
this.onFailure();
throw err;
}
}
private onSuccess() {
this.failures = 0;
if (this.state === CircuitState.HalfOpen) {
this.successes++;
if (this.successes >= this.options.successThreshold) {
this.state = CircuitState.Closed;
}
}
}
private onFailure() {
this.failures++;
this.lastFailureTime = Date.now();
if (this.failures >= this.options.failureThreshold) {
this.state = CircuitState.Open;
}
}
getState() { return this.state; }
}Attach one breaker per provider. Wrap the per-candidate call in invokeWithFallback:
// lib/llm/fallback.ts (updated)
import { CircuitBreaker, CircuitState } from "./circuit";
const breakers = new Map<string, CircuitBreaker>();
function getBreaker(provider: string): CircuitBreaker {
if (!breakers.has(provider)) {
breakers.set(provider, new CircuitBreaker());
}
return breakers.get(provider)!;
}
export async function invokeWithFallback(
candidates: FallbackCandidate[],
messages: BaseMessage[]
): Promise<FallbackResult> {
const errors: FallbackResult["errors"] = [];
for (const candidate of candidates) {
const breaker = getBreaker(candidate.provider);
if (breaker.getState() === CircuitState.Open) {
errors.push({
provider: candidate.provider,
modelName: candidate.modelName,
error: new Error("Circuit open"),
});
continue;
}
const policy = candidate.retryPolicy ?? DEFAULT_RETRY_POLICY;
try {
const response = await breaker.execute(() =>
invokeWithRetry(candidate.model, messages, policy, {
provider: candidate.provider,
modelName: candidate.modelName,
})
);
return { response, used: { provider: candidate.provider, modelName: candidate.modelName, attempt: 1 }, errors };
} catch (err: any) {
errors.push({ provider: candidate.provider, modelName: candidate.modelName, error: err });
}
}
const aggregate = new Error("All fallback candidates exhausted") as Error & { fallbacks: typeof errors };
aggregate.fallbacks = errors;
throw aggregate;
}Verify: Mock a provider to return 500 on every call. After five failures, the breaker opens. Subsequent calls skip that provider immediately. After the cooldown, a single success moves it to half-open; two successes close it.
Step 6: propagate cache-control and routing hints
When you sit behind a gateway that forwards provider headers (like x-ratelimit-remaining, retry-after, or cache-control directives), surface them to callers so they can make informed decisions. LangChain.js callbacks receive the raw response — hook into handleLLMEnd to extract headers.
// lib/llm/callbacks.ts
import { BaseCallbackHandler } from "@langchain/core/callbacks/base";
import { BaseMessage } from "@langchain/core/messages";
export interface GatewayHints {
rateLimitRemaining?: number;
rateLimitResetMs?: number;
cacheHit?: boolean;
providerLatencyMs?: number;
}
export class GatewayHintExtractor extends BaseCallbackHandler {
public hints: GatewayHints = {};
private startTime = 0;
handleLLMStart() {
this.startTime = Date.now();
}
handleLLMEnd(output: any, runId: string, parentRunId?: string, tags?: string[]) {
const generation = output.generations?.[0]?.[0];
const headers = generation?.message?.response_metadata?.headers ?? generation?.message?.additional_kwargs?.headers;
if (!headers) return;
this.hints = {
rateLimitRemaining: headers["x-ratelimit-remaining"] ? Number(headers["x-ratelimit-remaining"]) : undefined,
rateLimitResetMs: headers["x-ratelimit-reset"] ? Number(headers["x-ratelimit-reset"]) * 1000 : undefined,
cacheHit: headers["x-cache-status"] === "HIT",
providerLatencyMs: Date.now() - this.startTime,
};
}
}Attach the callback to each model instance. The hints object is available after invocation and can drive adaptive behavior — for example, skipping a provider when rateLimitRemaining === 0.
// lib/llm/chain.ts (updated)
import { GatewayHintExtractor } from "./callbacks";
export async function runWithResilience(prompt: string) {
const candidates = buildCandidates();
const messages = [new HumanMessage(prompt)];
// Attach extractors
const extractors = candidates.map(() => new GatewayHintExtractor());
candidates.forEach((c, i) => {
c.model.callbacks = [extractors[i]];
});
const result = await invokeWithFallback(candidates, messages);
// Attach hints from the successful candidate
const usedIdx = candidates.findIndex(c => c.provider === result.used.provider && c.modelName === result.used.modelName);
if (usedIdx >= 0) {
(result as any).gatewayHints = extractors[usedIdx].hints;
}
return result;
}Step 7: observe everything in production
Structured logs are your only debug tool at 2 AM. Emit one log line per attempt with these fields:
{
"timestamp": "2024-01-15T03:22:11.400Z",
"level": "warn",
"event": "llm_attempt_failed",
"provider": "openai",
"model": "gpt-4o",
"attempt": 2,
"maxAttempts": 3,
"statusCode": 429,
"retryable": true,
"retryAfterMs": 1200,
"circuitState": "closed",
"requestId": "req_abc123",
"traceId": "trace_xyz789"
}On success, log the chosen candidate, total attempts, latency, and gateway hints. On total failure, log the full fallbacks array. Correlate with your tracing system (OpenTelemetry, Datadog, etc.) using traceId.
// lib/llm/logging.ts
import { FallbackResult } from "./fallback";
export function logAttempt(
provider: string,
model: string,
attempt: number,
maxAttempts: number,
err: any,
circuitState: string,
requestId: string,
traceId: string
) {
console.warn(JSON.stringify({
timestamp: new Date().toISOString(),
level: "warn",
event: "llm_attempt_failed",
provider,
model,
attempt,
maxAttempts,
statusCode: err.statusCode,
retryable: err.retryable,
retryAfterMs: err.headers?.["retry-after"] ? parseRetryAfterMs(err.headers["retry-after"]) : undefined,
circuitState,
requestId,
traceId,
}));
}
export function logResult(result: FallbackResult, requestId: string, traceId: string) {
console.info(JSON.stringify({
timestamp: new Date().toISOString(),
level: "info",
event: "llm_invocation_succeeded",
provider: result.used.provider,
model: result.used.modelName,
totalAttempts: result.errors.length + 1,
fallbackCount: result.errors.length,
gatewayHints: (result as any).gatewayHints,
requestId,
traceId,
}));
}
export function logExhausted(errors: FallbackResult["errors"], requestId: string, traceId: string) {
console.error(JSON.stringify({
timestamp: new Date().toISOString(),
level: "error",
event: "llm_all_fallbacks_exhausted",
fallbacks: errors.map(e => ({
provider: e.provider,
model: e.modelName,
statusCode: e.error.statusCode,
message: e.error.message,
})),
requestId,
traceId,
}));
}
function parseRetryAfterMs(value: string): number | undefined {
const seconds = Number(value);
if (!isNaN(seconds)) return seconds * 1000;
const date = Date.parse(value);
if (!isNaN(date)) return Math.max(0, date - Date.now());
return undefined;
}Wire these into invokeWithRetry and invokeWithFallback. You now have a complete observability loop.
Step 8: test the full chain locally
Create a test harness that exercises every path without calling real APIs. Use msw or a simple mock server to simulate 429 → 500 → success sequences, circuit breaker transitions, and fallback ordering.
// tests/llm.resilience.test.ts
import { runWithResilience } from "../lib/llm/chain";
import { ChatOpenAI } from "@langchain/openai";
import { ChatAnthropic } from "@langchain/anthropic";
// Mock implementations that simulate failures
class MockChatModel {
constructor(private responses: Array<{ error?: Error; content?: string }>) {}
private callCount = 0;
callbacks: any[] = [];
async invoke(messages: any[]) {
const response = this.responses[this.callCount++] ?? { error: new Error("Exhausted") };
if (response.error) throw response.error;
return { content: response.content, response_metadata: { headers: {} } };
}
}
describe("resilience chain", () => {
it("retries twice then succeeds", async () => {
const model = new MockChatModel([
{ error: Object.assign(new Error("Rate limited"), { statusCode: 429, retryable: true }) },
{ error: Object.assign(new Error("Rate limited"), { statusCode: 429, retryable: true }) },
{ content: "success" },
]);
// Inject mock into candidates...
// Assert result.content === "success"
});
it("falls back to second provider when first exhausts", async () => {
const primary = new MockChatModel([
{ error: Object.assign(new Error("Down"), { statusCode: 503, retryable: true }) },
{ error: Object.assign(new Error("Down"), { statusCode: 503, retryable: true }) },
{ error: Object.assign(new Error("Down"), { statusCode: 503, retryable: true }) },
]);
const fallback = new MockChatModel([{ content: "fallback ok" }]);
// Inject and assert fallback used
});
it("opens circuit after threshold and skips provider", async () => {
// Configure breaker with failureThreshold: 2
// Call twice with 500, third call should skip immediately
});
});Run these in CI on every PR. They catch regressions when you upgrade LangChain.js or add new providers.
Step 9: tune policies per use case
Not all requests deserve the same resilience. A user-facing chat needs fast failover; a background summarization job can tolerate longer retries. Parameterize the policy at the call site:
// lib/llm/policies.ts
export const POLICIES = {
interactive: {
maxAttempts: 2,
baseDelayMs: 200,
maxDelayMs: 2000,
jitterFactor: 0.2,
fallbackCandidates: ["gpt-4o-mini", "claude-3-haiku"],
},
background: {
maxAttempts: 5,
baseDelayMs: 1000,
maxDelayMs: 30000,
jitterFactor: 0.5,
fallbackCandidates: ["gpt-4o", "claude-3-5-sonnet", "gemini-1.5-pro"],
},
critical: {
maxAttempts: 3,
baseDelayMs: 500,
maxDelayMs: 5000,
jitterFactor: 0.3,
fallbackCandidates: ["gpt-4o", "claude-3-5-sonnet", "gemini-1.5-pro", "gpt-4o-mini"],
},
} as const;Select the policy in your route handler:
// routes/chat.ts
import { POLICIES } from "../lib/llm/policies";
import { runWithResilience } from "../lib/llm/chain";
export async function handleChat(req: Request, res: Response) {
const policy = req.query.background ? POLICIES.background : POLICIES.interactive;
const result = await runWithResilience(req.body.prompt, policy);
res.json({ text: result.response.content, meta: result.used, hints: result.gatewayHints });
}Step 10: verify end-to-end in staging
Deploy to a staging environment with a canary release. Shadow 5% of production traffic to the new resilience layer. Compare these metrics against the old path:
- Success rate (should increase)
- P99 latency (may increase slightly due to retries; ensure it stays within SLO)
- Fallback rate (percentage of requests served by non-primary candidates)
- Circuit open events (should be rare; alert if > 1/minute)
- Cost per 1k tokens (fallbacks may use cheaper or more expensive models)
Roll back if P99 latency regresses > 20% or cost spikes unexpectedly. Once stable, promote to 100%.
You now have a production-grade LangChain.js error handling retry fallback stack: categorized failures, exponential backoff with Retry-After support, multi-provider fallback chains, provider-specific error normalization, circuit breaking, gateway hint propagation, structured observability, automated tests, and per-use-case policies. The code is framework-agnostic — drop it into any Node.js or TypeScript service using LangChain.js.