LlamaIndex retry logic fallback rate limits become critical the moment you move a RAG pipeline from a notebook to production traffic. A single 429 from your primary model shouldn’t abort a user query, and a hard dependency on one provider invites outages. This guide walks through building explicit retry and fallback handlers in LlamaIndex so rate limits and provider degradation degrade gracefully instead of throwing.
Step 1: Set up a minimal LlamaIndex project
Install the core package and an OpenAI-compatible LLM client. We’ll use the official OpenAI LLM adapter, but the pattern works for any LlamaIndex LLM implementation, including local models and gateways.
pip install llama-index openai tenacityLoad a few documents and build an in-memory vector index. This gives us a real query path to attach retries to.
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai import OpenAI
docs = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(docs)
primary_llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.llm = primary_llmIf you don’t have a data directory, create one with a single .txt file. The index build is unrelated to retries, but you need a query engine to exercise the failure path. Note that embedding generation during indexing also hits provider rate limits—if you use a hosted embedding model, wrap the from_documents call with the same retry discipline you apply to completions.
Step 2: Understand the failure modes you need to catch
OpenAI’s Python SDK raises openai.RateLimitError on HTTP 429 and openai.APIStatusError on other non-200 responses. When you point LlamaIndex at an OpenAI-compatible gateway, the same exception types surface because the underlying SDK is unchanged.
import openai
# Exception classes worth catching
rate_limit_err = openai.RateLimitError
status_err = openai.APIStatusError
connection_err = openai.APIConnectionErrorA robust LlamaIndex retry logic fallback rate limits strategy treats 429 as retryable with backoff, 5xx as retryable with backoff, and 4xx (except 429) as fatal for that provider. Connection errors are retryable but should count toward a separate timeout budget. Some providers and gateways return a Retry-After header on 429 responses; capturing it lets you wait the exact prescribed interval instead of guessing.
def get_retry_after(exc):
if hasattr(exc, "response") and exc.response is not None:
return exc.response.headers.get("Retry-After")
return NoneStep 3: Add exponential backoff to a single LLM call
Wrap the query in a function decorated with tenacity.retry. The decorator retries on rate-limit or connection errors, waiting 2 ** attempt seconds between tries. This is the simplest form of LlamaIndex retry logic.
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
@retry(
stop=stop_after_attempt(4),
wait=wait_exponential(multiplier=1, min=1, max=10),
retry=retry_if_exception_type((openai.RateLimitError, openai.APIConnectionError)),
reraise=True,
)
def bounded_query(engine, query_str):
return engine.query(query_str)
engine = index.as_query_engine()
response = bounded_query(engine, "What does the doc say about pricing?")The wait_exponential caps the wait at 10 seconds, so a poisoned provider won’t hang your request indefinitely. Set stop_after_attempt based on your end-user latency tolerance; four attempts with backoff rarely exceeds 15 seconds.
Respecting Retry-After
If you want to honor the server’s Retry-After instead of a fixed exponential curve, swap the wait strategy:
from tenacity import wait_base
class wait_retry_after(wait_base):
def __call__(self, retry_state):
exc = retry_state.outcome.exception()
ra = get_retry_after(exc)
return float(ra) if ra else 2 ** retry_state.attempt_number
@retry(stop=stop_after_attempt(4), wait=wait_retry_after(),
retry=retry_if_exception_type(openai.RateLimitError), reraise=True)
def bounded_query_respectful(engine, query_str):
return engine.query(query_str)Step 4: Implement multi-provider fallback in your LlamaIndex retry logic
Backoff alone fails when the provider is fully degraded. You need to switch models or providers. Define a list of LLMs ordered by preference and iterate over them, applying the retry wrapper per LLM.
llm_primary = OpenAI(model="gpt-4o-mini")
llm_secondary = OpenAI(model="gpt-3.5-turbo")
# A gateway such as n4n.ai exposes one OpenAI-compatible endpoint that addresses
# 240+ models and applies automatic fallback when a provider is rate-limited or
# degraded; we still keep client-side fallback for prompt-specific needs.
llm_gateway = OpenAI(model="auto", api_base="https://api.n4n.ai/v1", api_key="YOUR_KEY")
fallback_chain = [llm_primary, llm_secondary, llm_gateway]Now write the loop. For each LLM we set Settings.llm, rebuild the query engine (cheap for in-memory indexes), and run the bounded query. If all retries on a given LLM fail, move to the next.
def query_with_fallback(index, query_str, llms, max_attempts=4):
last_exc = None
for llm in llms:
Settings.llm = llm
engine = index.as_query_engine()
try:
return bounded_query(engine, query_str)
except Exception as e:
last_exc = e
print(f"LLM {llm.model} failed: {type(e).__name__}")
raise last_excFor cleaner integration, wrap the chain in an LLM subclass so LlamaIndex components can consume it transparently:
from llama_index.core.llms import LLM, ChatMessage, ChatResponse
from typing import List, Any
class FallbackLLM(LLM):
def __init__(self, llms: List[LLM]):
self.llms = llms
super().__init__()
def chat(self, messages: List[ChatMessage], **kwargs: Any) -> ChatResponse:
last = None
for llm in self.llms:
try:
return llm.chat(messages, **kwargs)
except Exception as e:
last = e
raise last
# complete() and streaming methods delegate similarly
def complete(self, prompt: str, **kwargs: Any):
last = None
for llm in self.llms:
try:
return llm.complete(prompt, **kwargs)
except Exception as e:
last = e
raise last
@property
def metadata(self):
return self.llms[0].metadataAssign Settings.llm = FallbackLLM(fallback_chain) and the rest of your LlamaIndex code stays unchanged.
Step 5: Verify the retry and fallback behavior
You can’t trust fallback code you haven’t watched fail. Replace the real LLM with a stub that raises on the first N calls, then succeeds.
from llama_index.core.llms import MockLLM
import openai
class FlakyMockLLM(MockLLM):
def __init__(self, fail_times=2):
self.fail_times = fail_times
self.calls = 0
def chat(self, messages, **kwargs):
self.calls += 1
if self.calls <= self.fail_times:
raise openai.RateLimitError("rate limit", response=None, body=None)
return super().chat(messages, **kwargs)
flaky = FlakyMockLLM(fail_times=2)
test_chain = [flaky, MockLLM()]
out = query_with_fallback(index, "test", test_chain)
assert out is not NoneIf the assertion passes, your LlamaIndex retry logic fallback rate limits handled the simulated 429s and then succeeded on the mock. In production, watch logs for the LLM ... failed lines to confirm failover is triggering as expected. If you route through a gateway that provides per-token usage metering (e.g., n4n.ai), tag fallback calls with a request ID so you can attribute cost precisely when a primary model deflects to a more expensive secondary.
A pytest suite with the flaky mock and a always-failing mock validates both retry exhaustion and chain advancement:
def test_fallback_chain():
always_fail = MockLLM()
always_fail.chat = lambda *a, **k: (_ for _ in ()).throw(openai.RateLimitError("x", None, None))
good = MockLLM()
chain = [always_fail, good]
res = query_with_fallback(index, "q", chain)
assert res is not NoneStep 6: Production hardening
Retries amplify load during incidents. Add a circuit breaker that stops calling a provider after consecutive failures, and set request_timeout on the OpenAI client to avoid stuck sockets. When using an OpenAI-compatible gateway, honor its cache-control hints by passing extra_headers so repeated identical prompts hit provider caches instead of triggering new rate limits.
Keep your fallback chain short. Three providers is enough for most teams; more adds latency and obscures which path actually served the answer. Instrument each attempt with timing so you can see backoff sleeps in your traces. For streaming responses, apply the same try/except around stream_chat and rebuild the generator on failure rather than mid-stream.
The code above is deliberately minimal. Drop it into a service layer behind your API, and your RAG endpoints will survive provider hiccups without a rewrite.