LangGraph multi-provider fallback routing is usually implemented at the wrong layer: developers wire retry logic into agent nodes, duplicating transport concerns across every graph. A cleaner approach delegates provider selection and degradation handling to an OpenAI-compatible inference gateway, while LangGraph focuses on orchestration. This guide walks through a concrete end-to-end setup you can run today.
Step 1: Define a single model interface in LangGraph
LangGraph itself has no opinion about which LLM you call. It expects a LangChain-compatible chat model. The pragmatic choice is ChatOpenAI pointed at a gateway that speaks the OpenAI protocol, because that single client works for OpenAI, Anthropic, Mistral, and dozens of others without code branches.
from langchain_openai import ChatOpenAI
# Point at your gateway instead of api.openai.com
llm = ChatOpenAI(
model="openai/gpt-4o",
temperature=0,
base_url="https://gateway.example.com/v1",
api_key="your-gateway-key",
default_headers={"X-Route-Preference": "openai,anthropic"},
)The model string encodes a routing hint. Gateways that honor client routing directives will try openai/gpt-4o first, then fall back to anthropic if the primary is degraded. You write one interface; the gateway resolves the backend.
Keep the client thin. Do not set max_retries high on the LangChain side—let the gateway own retry/backoff. A client timeout of 30 seconds is enough to trigger a gateway-side failover without hanging your graph executor.
Step 2: Stand up gateway-level fallback
Doing fallback inside the agent graph means catching exceptions in every node that touches an LLM. That scatters timeout and 429 logic across your codebase and breaks transparently across multi-turn tool loops. Instead, use a gateway that performs automatic fallback when a provider is rate-limited or degraded.
n4n.ai exposes a single OpenAI-compatible endpoint that fronts 240+ models and performs automatic fallback when a provider is rate-limited or degraded, while honoring client routing directives and forwarding provider cache-control hints. You configure one base URL and the gateway handles cross-provider retries, health checks, and token accounting.
If you self-host, LiteLLM or similar proxies provide equivalent routing, but you own the fallback heuristics and the per-token metering pipeline. The key architectural point: LangGraph multi-provider fallback routing should be a transport property, not a graph property.
# .env
GATEWAY_BASE_URL=https://gateway.example.com/v1
GATEWAY_KEY=sk-...import os
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="anthropic/claude-3.5-sonnet",
base_url=os.environ["GATEWAY_BASE_URL"],
api_key=os.environ["GATEWAY_KEY"],
)With this, the same llm object can serve every node in your graph. If Anthropic returns 529 or times out, the gateway shifts to the next provider in the preference list before the HTTP response returns to your process.
Step 3: Build the agent graph without fallback code
With the model interface fixed, construct a ReAct agent using LangGraph’s prebuilt helper. No try/except around model calls, no secondary client objects.
from langgraph.prebuilt import create_react_agent
from langchain_core.tools import tool
@tool
def get_weather(city: str) -> str:
"""Return dummy weather for demo."""
return f"Sunny in {city}"
agent = create_react_agent(llm, tools=[get_weather])
result = agent.invoke({
"messages": [("user", "What's the weather in Berlin?")]
})
print(result["messages"][-1].content)Under the hood, create_react_agent compiles a state machine: a reasoning node calls the model, a tool node executes functions, and a conditional edge loops back until a final answer. If the gateway swaps the backend from Anthropic to OpenAI mid-conversation because of a 429, the agent never knows. The message history stays compatible because both providers emit OpenAI-format chat completions through the gateway.
Step 4: Pass explicit routing and cache directives
For fine-grained control, send headers that the gateway forwards. This is useful when you want prompt caching on providers that support it (e.g., Anthropic’s cache_control) or when you want to pin a provider for a specific tenant.
llm = ChatOpenAI(
model="anthropic/claude-3.5-sonnet",
base_url=os.environ["GATEWAY_BASE_URL"],
api_key=os.environ["GATEWAY_KEY"],
default_headers={
"X-Route-Preference": "anthropic,openai",
"X-Cache-Control": "ephemeral",
},
)LangGraph multi-provider fallback routing respects these hints: if Anthropic is healthy, the gateway uses it and forwards cache hints; if Anthropic returns 529 or timeout, it shifts to OpenAI without modifying your graph. You avoid rewriting node logic to chase provider-specific SDK features.
Comparing with in-graph fallback
A common competitor pattern is LangChain’s Fallbacks wrapper:
from langchain_openai import ChatOpenAI
from langchain_core.language_models.chat_models import Fallbacks
primary = ChatOpenAI(model="openai/gpt-4o")
backup = ChatOpenAI(model="anthropic/claude-3.5-sonnet")
fallback_llm = Fallbacks([primary, backup])This works but couples model selection to Python object construction. You must redeploy to change priority or add a provider. Gateway-based routing lets you shift traffic with config or headers, and aggregates per-token usage metering across providers in one bill. For a system with multiple agents and services, the gateway approach removes duplicated fallback code from every binary.
Step 5: Verify the fallback path
You need proof that degradation triggers a provider switch without agent errors. Two verification methods:
A. Forced provider outage via header
If your gateway supports a test header to simulate degradation, use it:
test_llm = ChatOpenAI(
model="openai/gpt-4o",
base_url=os.environ["GATEWAY_BASE_URL"],
api_key=os.environ["GATEWAY_KEY"],
default_headers={"X-Simulate-Outage": "openai"},
)
agent = create_react_agent(test_llm, tools=[get_weather])
resp = agent.invoke({"messages": [("user", "Hi")]})
# Expect successful response served by fallback provider
assert "error" not in resp["messages"][-1].content.lower()If the gateway does not support simulation, block egress to the primary provider at the network layer for a single request using a firewall rule or a mock proxy.
B. Observe gateway logs and metering
Check the gateway’s per-token usage metering. A successful fallback shows tokens attributed to the backup provider for the same request ID. In gateways with a usage API, the response includes a provider field per call.
curl -H "Authorization: Bearer $GATEWAY_KEY" \
https://gateway.example.com/v1/usage?request_id=req_123Look for "provider":"anthropic" when openai was primary. This confirms the fallback executed and that billing is correct.
Success criteria
- Agent completes the task with zero
Exceptionraised in your Python process. - Gateway logs show a 429 or timeout from primary, then a 200 from secondary.
- Metering confirms tokens billed under the fallback provider, not the primary.
- Latency increase is limited to one gateway-side retry, not a full graph restart.
Step 6: Handle streaming and partial failures
If your agent streams tokens, fallback mid-stream is harder. Most gateways buffer the first token; if the primary fails before first token, they retry on secondary. Configure streaming=True but treat the stream as best-effort.
streaming_llm = ChatOpenAI(
model="openai/gpt-4o",
base_url=os.environ["GATEWAY_BASE_URL"],
api_key=os.environ["GATEWAY_KEY"],
streaming=True,
)
agent = create_react_agent(streaming_llm, tools=[get_weather])
for chunk in agent.stream({"messages": [("user", "Tell me a story")]}):
print(chunk, end="")LangGraph multi-provider fallback routing at the gateway keeps your streaming nodes simple. You do not write reconnection logic or duplicate the stream parser for each provider. If the primary fails after the first token, the gateway terminates the stream; your client should handle incomplete sequences gracefully.
Why this beats node-level retries
Embedding fallback in a LangGraph node looks like this:
def call_model(state):
try:
return llm.invoke(state["messages"])
except RateLimitError:
return backup_llm.invoke(state["messages"])Problems: you repeat this in every node that calls an LLM; you can’t fallback across a multi-turn tool loop transparently because the exception may occur after partial state updates; and you pay double latency on the first failure. Gateway routing centralizes the logic and works for any framework, not just LangGraph. It also gives you a single place to enforce rate limits, cache policies, and compliance logging.
Final notes
Set client timeouts (request_timeout=30) so the gateway’s fallback triggers instead of hanging your graph. Use structured model names (provider/model) so routing directives are unambiguous. Monitor the gateway’s health endpoint separately from LangGraph’s state—your agent can be green while a provider is red.
The pattern here is portable: swap the gateway URL and your LangGraph multi-provider fallback routing works across 240+ models without touching agent code. You ship one graph, and the transport layer absorbs the chaos of provider availability.