LangChain retry vs fallback: which strategy to use

LangChain retry vs fallback comparison comes up whenever you move past toy projects and need production-grade reliability. Retry handles transient failures on a single model; fallback switches to a different model when the primary fails or degrades. They solve different problems, compose differently, and have distinct cost and latency profiles. Understanding both lets you build routing logic that survives provider outages without blowing your token budget.

What retry does in LangChain

LangChain’s Runnable.with_retry wraps any runnable — chat model, chain, or custom component — and re-executes it on failure. You configure stop conditions, wait strategies, and exception filters. The default retries on any exception with exponential backoff, which is rarely what you want in production.

from langchain_openai import ChatOpenAI
from langchain_core.runnables import RunnableConfig

llm = ChatOpenAI(model="gpt-4o-mini")

retry_llm = llm.with_retry(
    stop_after_attempt=3,
    wait_exponential_jitter=True,
    retry_if_exception_type=(RateLimitError, APIConnectionError, InternalServerError),
)

Key behaviors: retry preserves the exact same model and parameters across attempts. Input tokens are re-sent each attempt, so you pay for them multiple times. Latency compounds — a 3-attempt retry with exponential backoff can add 10-30 seconds to a request. The runnable interface stays identical, so downstream code doesn’t change.

Retry shines for transient provider issues: brief rate-limit spikes, momentary network blips, or 5xx errors that resolve quickly. It fails for model-level problems: context window exceeded, content policy rejections, or sustained provider degradation.

What fallback does in LangChain

Runnable.with_fallbacks accepts a list of alternative runnables and tries them sequentially until one succeeds. Each fallback can be a completely different model, provider, or even a different chain architecture.

from langchain_anthropic import ChatAnthropic
from langchain_google_genai import ChatGoogleGenerativeAI

primary = ChatOpenAI(model="gpt-4o")
fallback_1 = ChatAnthropic(model="claude-3-5-sonnet-20241022")
fallback_2 = ChatGoogleGenerativeAI(model="gemini-1.5-pro")

chain = primary.with_fallbacks([fallback_1, fallback_2])

Fallbacks execute only when the previous runnable raises an exception. Successful responses short-circuit the chain. Each fallback receives the original input — no automatic prompt translation, so you must ensure prompts work across models. Token costs accrue only for the successful attempt (plus any failed attempts that consumed tokens before erroring).

Fallback handles model-level failures: context overflow, capability gaps, provider outages, or cost optimization. It introduces routing complexity — you need to understand each model’s strengths, token limits, and failure modes.

Comparison across dimensions

Capabilities

Retry operates on a single model. It cannot recover from “context length exceeded” or “model refuses this request” because the same constraints apply every attempt. Fallback can route to a model with a larger context window, different safety filters, or specialized capabilities (code vs creative vs reasoning).

Fallback chains can mix providers — OpenAI → Anthropic → Google → local model. This requires managing multiple API keys, rate limits, and response formats. LangChain normalizes the BaseChatModel interface, but system prompts, tool calling schemas, and output styles still differ.

Price and cost model

Retry multiplies input token costs. A 3-attempt retry on a 10k-token prompt bills 30k input tokens even if only the third succeeds. Output tokens bill once (successful attempt). At gpt-4o pricing, a 10k/2k request retried 3x costs ~$0.015 extra in input tokens.

Fallback bills only the successful path’s tokens (assuming failed attempts error before token generation). However, you pay for the most expensive model in the chain if it succeeds. A gpt-4o → claude-3.5-sonnet fallback that lands on Sonnet costs Sonnet rates, not gpt-4o rates.

Neither strategy optimizes for cost automatically. For cost-aware routing, you need a router that selects models based on task complexity — a separate pattern from both retry and fallback.

Latency and throughput

Retry adds variable latency. Exponential backoff with jitter (the recommended default) means: attempt 1 fails → wait ~1s → attempt 2 fails → wait ~2s → attempt 3 succeeds. Total added latency: 3-5 seconds typical, up to 30s under heavy backoff. Throughput drops because the thread/connection is occupied during backoff.

Fallback adds deterministic latency per hop. Each fallback attempt incurs full model latency. OpenAI → Anthropic fallback adds ~1-2s for the Anthropic call if OpenAI fails. No backoff waiting unless you wrap each fallback in its own retry.

Under sustained load, retry consumes more connection pool slots (held during backoff). Fallback releases the primary connection immediately on failure, acquiring the next provider’s connection.

Ergonomics and debugging

Retry is simpler to add — one wrapper, same interface. Debugging means checking retry count in logs or callbacks. LangChain’s RunnableConfig exposes metadata with attempt number if you configure it.

Fallback requires maintaining a prioritized model list. Debugging means tracing which model succeeded. LangChain’s callback system emits on_chain_start/on_chain_end for each fallback, but you need structured logging to correlate attempts.

Both integrate with LangSmith tracing. Retry shows as nested spans under one run. Fallback shows as sibling spans — cleaner for debugging which model actually responded.

Ecosystem and limits

Retry works with any Runnable — chains, tools, custom functions. Limits: max attempts (practical ceiling ~5), backoff ceiling, exception filtering granularity. You cannot retry only specific error codes from provider responses without custom exception classes.

Fallback works with any Runnable list. Limits: no built-in health checks (tries dead models repeatedly), no circuit breaker, no load balancing. Order is static — no dynamic reordering based on latency or error rates. For advanced routing (weighted, latency-aware, cost-aware), you need a separate router component.

Both are stateless. Neither preserves conversation context across retries or fallbacks automatically — that’s your responsibility via message history management.

Composing retry and fallback

They compose. The practical pattern: wrap each model in retry, then chain fallbacks.

primary = ChatOpenAI(model="gpt-4o").with_retry(
    stop_after_attempt=2,
    retry_if_exception_type=(RateLimitError, APIConnectionError),
)
fallback_1 = ChatAnthropic(model="claude-3-5-sonnet-20241022").with_retry(
    stop_after_attempt=2,
    retry_if_exception_type=(RateLimitError, APIConnectionError),
)
fallback_2 = ChatGoogleGenerativeAI(model="gemini-1.5-pro").with_retry(
    stop_after_attempt=1,  # last resort, fail fast
)

robust_chain = primary.with_fallbacks([fallback_1, fallback_2])

This handles transient errors on each provider (retry) and provider-level failures (fallback). Each model gets 2 retries before falling through. Total worst-case: 2 primary + 2 fallback_1 + 1 fallback_2 = 5 model calls.

You can also nest: a fallback chain as one branch of a router, with retry on the router itself. The composition is flexible because both return Runnable.

Which to choose

Use retry when

• Transient provider errors dominate: Rate limits, brief network issues, 5xx spikes that resolve in seconds.
• Single model is required: Compliance, consistency, or contract mandates one specific model.
• Cost predictability matters: You know the exact model and max token multiplier.
• Latency budget is tight: Fallback adds full model latency; retry adds only backoff wait (often shorter than a second model call).

Use fallback when

• Model capability gaps exist: Primary model hits context limits, refuses valid requests, or lacks a capability (vision, tool calling, specific language).
• Provider outage risk is real: You need continuity when OpenAI/Anthropic/Google goes down.
• Cost optimization across models: Route simple tasks to cheaper models, complex to expensive — though this needs a router, not static fallback.
• Degraded quality is acceptable: Fallback model may produce different style/quality; you accept variance for availability.

Use both (the production default)

Wrap each model in retry, then chain fallbacks. This covers the full failure spectrum:

1. Transient error on primary → retry succeeds (fastest recovery)
2. Transient errors exhaust retries → fallback to secondary → retry succeeds
3. Secondary exhausted → tertiary → …
4. All exhausted → surface error to caller

This pattern adds ~2-5 lines per model and handles 95% of production failure modes.

Avoid both when

• You need semantic routing: “Use gpt-4o for coding, claude for writing, gemini for long context.” That’s a router, not fallback.
• You need circuit breaking: Stop sending traffic to a degraded provider for a cooldown period. Neither retry nor fallback implements this.
• You need load balancing: Distribute traffic across healthy providers. Fallback is sequential, not parallel.
• You need cost-aware model selection: Requires task classification + model pricing matrix + latency SLA.

Practical recommendations

Start with retry on your primary model. Configure retry_if_exception_type narrowly — only RateLimitError, APIConnectionError, InternalServerError. Never retry BadRequestError (context overflow, invalid prompt) or AuthenticationError.

Add one fallback to a different provider. Different provider = independent failure domain. OpenAI + Anthropic is the standard pair; add Google or a local model as tertiary.

Set stop_after_attempt=2 on primary, 1 on fallbacks. Fail fast on fallbacks — they’re your last resort.

Log which model succeeded. Add a callback that records model_used in your observability. You’ll need this for debugging and cost allocation.

Test failure injection. Kill the primary API key, verify fallback triggers. Simulate rate limits, verify retry backs off. Automate this in CI.

If you run multiple models across providers and want a single endpoint that handles routing, fallback, and per-token metering without building the orchestration yourself, n4n.ai exposes one OpenAI-compatible endpoint addressing 240+ models with automatic fallback when a provider is rate-limited or degraded — but the patterns above work regardless of infrastructure.

Summary

Retry and fallback are orthogonal reliability primitives. Retry handles “try again on the same model.” Fallback handles “try a different model.” Compose them: retry each model, fallback across models. Configure narrowly, log explicitly, test failure paths. That’s the pattern that survives production.