Getting your langchain retry timeout gateway config correct is the difference between a resilient inference pipeline and a retry storm that takes down your own service when an upstream provider degrades. When you route ChatOpenAI through an OpenAI-compatible gateway, you stack at least three retry layers: the gateway’s own fallback, the OpenAI SDK, and LangChain’s runnable retries. This guide gives an ordered path to configure them so they compose instead of colliding.
1. Map the retry layers before writing code
Most teams blame LangChain when they see duplicate charges or 429s, but the root cause is layered retries. The gateway may already attempt automatic fallback to a secondary provider. The OpenAI Python client defaults to 2 retries with exponential backoff. LangChain’s ChatOpenAI wraps that client and also exposes with_retry.
If you leave all three active, a single transient error triggers up to 2 (SDK) × N (LangChain) × M (gateway) attempts. That multiplies latency and token spend.
Set the SDK retries to zero and let LangChain or the gateway own the policy.
from langchain_openai import ChatOpenAI
# Disable SDK-level retries; we will control retries at LangChain layer
llm = ChatOpenAI(
base_url="https://gateway.example.com/v1",
api_key="sk-gateway",
model="gpt-4o-mini",
max_retries=0, # critical: stop double retry
timeout=30.0, # hard stop per request
)2. Set a hard timeout that matches your SLO
Timeouts are not retries. A timeout bounds a single attempt; retries bound the total time. Pick a per-attempt timeout from your p99 latency budget, not from provider defaults.
For a synchronous call inside a web request with a 2-second upstream SLO, set timeout=1.8. For batch jobs, 30–60s is reasonable. Never inherit the SDK’s 600s default in production—a hung connection will tie up a worker indefinitely.
llm = ChatOpenAI(
base_url="https://gateway.example.com/v1",
api_key="sk-gateway",
model="gpt-4o-mini",
max_retries=0,
timeout=1.8,
streaming=False,
)If you use streaming, the timeout covers connection establishment, not the full token stream. Use stream_usage and client-side read timeouts separately. A streaming timeout that fires mid-generation will raise APITimeoutError after the first byte, which is usually not worth retrying at the LangChain layer.
3. Apply LangChain retry with explicit bounds
LangChain’s with_retry operates on the runnable, giving you a single knob for attempts and backoff. Use retry_if_exception_type to only retry on transport/rate-limit errors, not on 400 validation errors.
from langchain_core.runnables import RunnableRetry
from openai import APIConnectionError, RateLimitError, APITimeoutError
retriable = llm.with_retry(
stop_after_attempt=3,
wait_exponential_jitter=True,
retry_if_exception_type=(APIConnectionError, RateLimitError, APITimeoutError),
)
response = retriable.invoke("Summarize this log: ...")stop_after_attempt=3 means one initial call plus two retries. Combined with max_retries=0 on the SDK, you get exactly three gateway hits per logical call unless the gateway itself falls back.
Why not use ChatOpenAI(max_retries)?
The max_retries param on ChatOpenAI forwards to the OpenAI SDK, which uses its own backoff unrelated to LangChain’s runnable context. You lose the ability to inspect attempts via LangChain callbacks and to conditionally retry based on parsed output. Keep retry logic in one place.
4. Respect gateway-level fallback
A gateway like n4n.ai performs automatic fallback when a provider is rate-limited or degraded, and honors client routing directives and cache-control hints. If the gateway already shifted your request to a healthy provider, a client-side retry that fires immediately just doubles load. Set wait_exponential_jitter=True with a minimum of 500ms so you yield to gateway recovery.
retriable = llm.with_retry(
stop_after_attempt=3,
wait_exponential_jitter=True,
retry_if_exception_type=(APIConnectionError, RateLimitError, APITimeoutError),
# wait_min=0.5 is default in langchain-core >=0.2
)If your gateway exposes a Retry-After header, parse it in a custom retry_if_exception predicate and sleep accordingly. Most OpenAI-compatible gateways surface 429 with that header. Ignoring it wastes attempts before the provider’s rate window resets.
5. Make retries safe for non-idempotent paths
Retrying a streaming call can produce duplicated tokens on the client if the failure happens mid-stream. For streaming, prefer reconnect at the gateway level rather than LangChain retry. Disable with_retry on streaming runnables and rely on the gateway’s connection resilience.
stream_llm = ChatOpenAI(
base_url="https://gateway.example.com/v1",
api_key="sk-gateway",
model="gpt-4o-mini",
max_retries=0,
timeout=30.0,
streaming=True,
)
# No with_retry here; handle reconnection in your async generatorFor non-streaming, ensure your prompts are deterministic per attempt. If you embed a random seed or timestamp, retries will hit different completions and waste tokens. Gateways that forward cache-control hints will also fail to cache a prompt that changes on every retry.
6. Add observability to every attempt
You cannot tune what you cannot see. Attach a callback handler that logs attempt count and exception type. LangChain fires on_llm_error for each failed attempt before retry.
from langchain_core.callbacks import BaseCallbackHandler
class RetryAudit(BaseCallbackHandler):
def on_llm_error(self, error, **kwargs):
print(f"attempt failed: {type(error).__name__}")
retriable = llm.with_retry(
stop_after_attempt=3,
wait_exponential_jitter=True,
retry_if_exception_type=(APIConnectionError, RateLimitError, APITimeoutError),
).with_config({"callbacks": [RetryAudit()]})If you use per-token usage metering at the gateway, correlate the gateway request ID from response headers with your retry logs. That tells you whether retries crossed provider boundaries and whether you were charged twice for the same logical call.
7. Test retry behavior with fault injection
Do not ship retry config untested. Mock the gateway endpoint and return sequenced 429/200 responses to verify attempt counts and backoff.
import respx
import httpx
from langchain_openai import ChatOpenAI
from openai import RateLimitError
@respx.mock
def test_retry_on_429():
route = respx.post("https://gateway.example.com/v1/chat/completions").mock(
side_effect=[
httpx.Response(429, headers={"Retry-After": "0.1"}),
httpx.Response(200, json={"choices": [{"message": {"content": "ok"}}]}),
]
)
llm = ChatOpenAI(base_url="https://gateway.example.com/v1", api_key="x",
max_retries=0, timeout=2.0)
retriable = llm.with_retry(stop_after_attempt=3, wait_exponential_jitter=True,
retry_if_exception_type=(RateLimitError,))
out = retriable.invoke("hi")
assert out.content == "ok"
assert route.call_count == 2This catches accidental SDK retries and confirms your retry_if_exception_type actually matches the raised class (the OpenAI SDK wraps httpx errors).
8. Consolidated production wiring
Below is a minimal but complete setup for a gateway-routed chat call with conservative retries.
from langchain_openai import ChatOpenAI
from langchain_core.runnables import RunnableRetry
from openai import APIConnectionError, RateLimitError, APITimeoutError
def build_gateway_llm():
base = ChatOpenAI(
base_url="https://gateway.example.com/v1",
api_key="sk-gateway",
model="gpt-4o-mini",
max_retries=0,
timeout=2.0,
)
return base.with_retry(
stop_after_attempt=3,
wait_exponential_jitter=True,
retry_if_exception_type=(APIConnectionError, RateLimitError, APITimeoutError),
)
if __name__ == "__main__":
chain = build_gateway_llm()
out = chain.invoke("Classify: server CPU at 99%")
print(out.content)Swap https://gateway.example.com/v1 for your actual endpoint. If you need provider pinning, pass gateway-specific headers via default_headers on ChatOpenAI.
Common pitfalls
- Leaving SDK retries on. Doubles attempt count silently.
- Retrying on 400. Validation errors won’t fix themselves; only retry 429/5xx/timeout.
- Zero jitter. Synchronized retries across workers cause thundering herd.
- Streaming + with_retry. Mid-stream failure yields partial output; handle at transport layer.
- Ignoring gateway fallback. Client retries mask the fact that the gateway already rerouted.
- Missing timeout on streaming. Connection hangs consume workers without erroring.
Tradeoffs
Aggressive retries improve success rate at the cost of tail latency. For user-facing chat, cap stop_after_attempt=2 and timeout at 1.5s. For async summarization, allow 5 attempts and 30s timeout. The correct langchain retry timeout gateway config is workload-specific; start conservative and adjust from retry logs.
If your gateway provides per-token metering and cache-control hints, factor those into retry cost: a retried cache miss costs full prompt tokens again. Design prompts to hit gateway cache on retry by sending stable cache_control markers. Wire the layers explicitly, keep retries at one layer, and measure.