n4nAI
Open console →
Framework tutorialsHow-toOne Backend, Every Model: Swapping GPT-5, Claude, Gemini & Llama Across Frameworks

Switching LLM providers in LlamaIndex in five minutes

Learn to switch LLM providers in LlamaIndex with a unified endpoint — swap GPT-5, Claude, Gemini, and Llama models in minutes without rewriting your application code.

n4n Team3 min read569 words

Audio narration

Coming soon — every post will get a voice note here.

Most LlamaIndex tutorials hardcode a single provider. In production, you need to swap models — GPT-5 for reasoning, Claude for long context, Gemini for cost, Llama for privacy — without rewriting your indexes, agents, or query pipelines. This guide shows how to switch llm provider in llamaindex using a single OpenAI-compatible endpoint that routes to 240+ models, handles fallbacks automatically, and surfaces per-token usage. You’ll have a working multi-provider setup in five minutes.

Step 1: Install the dependencies

LlamaIndex’s OpenAI integration works with any OpenAI-compatible API. Install the core package and the LLM abstraction:

pip install llama-index llama-index-llms-openai

If you’re using a specific framework integration (agents, workflows, RAG), add those too:

pip install llama-index-agent-openai llama-index-workflows

Step 2: Configure the unified endpoint

Instead of managing separate API keys, base URLs, and client configurations for each provider, point LlamaIndex at one endpoint that handles routing, fallbacks, and usage metering. Set these environment variables:

export OPENAI_API_KEY="n4n-your-key-here"
export OPENAI_API_BASE="https://api.n4n.ai/v1"

The endpoint accepts standard OpenAI headers plus routing directives. For example, to force a specific model:

import os
from llama_index.llms.openai import OpenAI

llm = OpenAI(
    model="gpt-5",  # or "claude-3-5-sonnet", "gemini-1.5-pro", "llama-3.1-405b"
    api_key=os.getenv("OPENAI_API_KEY"),
    api_base=os.getenv("OPENAI_API_BASE"),
    temperature=0.1,
    max_tokens=4096,
)

You can also pass routing hints at request time using extra headers:

from llama_index.core.llms import ChatMessage

response = llm.chat(
    messages=[ChatMessage(role="user", content="Summarize this document")],
    extra_headers={
        "x-n4n-model": "claude-3-5-sonnet",  # override per request
        "x-n4n-fallback": "gpt-5,gemini-1.5-pro",  # fallback chain
    }
)

This is the core of how to switch llm provider in llamaindex without code changes — the model identifier becomes a runtime parameter, not a deployment decision.

Step 3: Wire the LLM into your LlamaIndex pipeline

LlamaIndex uses a global Settings object or per-component injection. For most applications, set it globally once at startup:

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
import os

Settings.llm = OpenAI(
    model="gpt-5",
    api_key=os.getenv("OPENAI_API_KEY"),
    api_base=os.getenv("OPENAI_API_BASE"),
    temperature=0.1,
)

# Optional: configure embedding model separately
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-large",
    api_key=os.getenv("OPENAI_API_KEY"),
    api_base=os.getenv("OPENAI_API_BASE"),
)

Now every index, query engine, agent, and workflow uses this LLM. To swap providers for a specific component, instantiate a separate LLM instance:

from llama_index.core import VectorStoreIndex
from llama_index.llms.openai import OpenAI

# Long-context model for this specific index
claude_llm = OpenAI(
    model="claude-3-5-sonnet",
    api_key=os.getenv("OPENAI_API_KEY"),
    api_base=os.getenv("OPENAI_API_BASE"),
    max_tokens=8192,
)

index = VectorStoreIndex.from_documents(documents, llm=claude_llm)
query_engine = index.as_query_engine(llm=claude_llm)

Step 4: Implement model selection logic

Hardcoding model names defeats the purpose. Build a small resolver that picks the right model for the task:

# model_resolver.py
from enum import Enum
from dataclasses import dataclass
from llama_index.llms.openai import OpenAI
import os

class ModelTier(Enum):
    REASONING = "gpt-5"
    LONG_CONTEXT = "claude-3-5-sonnet"
    COST_EFFICIENT = "gemini-1.5-flash"
    PRIVATE = "llama-3.1-405b"

@dataclass
class ModelConfig:
    tier: ModelTier
    temperature: float = 0.1
    max_tokens: int = 4096

    def build_llm(self) -> OpenAI:
        return OpenAI(
            model=self.tier.value,
            api_key=os.getenv("OPENAI_API_KEY"),
            api_base=os.getenv("OPENAI_API_BASE"),
            temperature=self.temperature,
            max_tokens=self.max_tokens,
        )

# Usage
def get_llm_for_task(task_type: str) -> OpenAI:
    mapping = {
        "code_generation": ModelTier.REASONING,
        "document_analysis": ModelTier.LONG_CONTEXT,
        "high_volume_chat": ModelTier.COST_EFFICIENT,
        "pii_processing": ModelTier.PRIVATE,
    }
    tier = mapping.get(task_type, ModelTier.REASONING)
    return ModelConfig(tier=tier).build_llm()

Wire this into your query engine factory:

from llama_index.core import VectorStoreIndex
from model_resolver import get_llm_for_task

def build_query_engine(documents, task_type: str):
    llm = get_llm_for_task(task_type)
    index = VectorStoreIndex.from_documents(documents, llm=llm)
    return index.as_query_engine(llm=llm, similarity_top_k=5)

Step 5: Add observability and usage tracking

You need to know which model served each request and what it cost. The unified endpoint returns usage in the standard OpenAI response format, plus provider metadata. Hook into LlamaIndex’s callback system:

from llama_index.core import Settings
from llama_index.core.callbacks import CallbackManager, TokenCountingHandler
import tiktoken

token_counter = TokenCountingHandler(
    tokenizer=tiktoken.encoding_for_model("gpt-4").encode,
    verbose=False,
)

Settings.callback_manager = CallbackManager([token_counter])

# After running queries:
print(f"Prompt tokens: {token_counter.prompt_llm_token_count}")
print(f"Completion tokens: {token_counter.completion_llm_token_count}")
print(f"Total tokens: {token_counter.total_llm_token_count}")

For per-request provider metadata, use the raw response from the LLM call:

from llama_index.core.llms import ChatMessage

response = llm.chat(
    messages=[ChatMessage(role="user", content="Test")],
    extra_headers={"x-n4n-model": "claude-3-5-sonnet"},
)

# The raw response includes provider info
print(response.raw)  # Contains provider, model, latency, cache status

The raw field surfaces the provider’s response headers including x-n4n-provider, x-n4n-model, x-n4n-latency-ms, and cache-control hints — useful for dashboards and alerting.

Step 6: Verify the setup end to end

Run this verification script to confirm everything works:

# verify.py
import os
from llama_index.core import Settings, VectorStoreIndex, Document
from llama_index.llms.openai import OpenAI
from model_resolver import get_llm_for_task, ModelTier

# 1. Verify global settings
Settings.llm = OpenAI(
    model="gpt-5",
    api_key=os.getenv("OPENAI_API_KEY"),
    api_base=os.getenv("OPENAI_API_BASE"),
)

# 2. Test direct chat
response = Settings.llm.complete("Say 'provider switch verified' exactly")
assert "provider switch verified" in response.text.lower()
print("✓ Global LLM works")

# 3. Test per-task resolver
for tier in ModelTier:
    llm = ModelConfig(tier=tier).build_llm()
    resp = llm.complete(f"Confirm you are {tier.value}")
    assert tier.value in resp.text.lower() or "confirm" in resp.text.lower()
    print(f"✓ {tier.value} responds")

# 4. Test RAG pipeline with model swap
docs = [Document(text="LlamaIndex routes to 240+ models via a single endpoint.")]
engine = build_query_engine(docs, "document_analysis")
result = engine.query("How many models?")
assert "240" in str(result)
print("✓ RAG pipeline works with long-context model")

# 5. Test fallback behavior (requires network)
# This will route through the fallback chain if primary is degraded
fallback_llm = OpenAI(
    model="gpt-5",
    api_key=os.getenv("OPENAI_API_KEY"),
    api_base=os.getenv("OPENAI_API_BASE"),
)
resp = fallback_llm.complete(
    "Test",
    extra_headers={"x-n4n-fallback": "claude-3-5-sonnet,gemini-1.5-pro"}
)
print(f"✓ Fallback chain accepted, provider: {resp.raw.get('headers', {}).get('x-n4n-provider')}")

print("\nAll checks passed. You can switch llm provider in llamaindex at runtime.")

Run it:

python verify.py

Expected output:

✓ Global LLM works
✓ gpt-5 responds
✓ claude-3-5-sonnet responds
✓ gemini-1.5-flash responds
✓ llama-3.1-405b responds
✓ RAG pipeline works with long-context model
✓ Fallback chain accepted, provider: anthropic

All checks passed. You can switch llm provider in llamaindex at runtime.

Step 7: Handle streaming and async patterns

Production workloads need streaming. The unified endpoint supports SSE streaming identically to OpenAI:

import asyncio
from llama_index.llms.openai import OpenAI

async def stream_response(prompt: str, model: str):
    llm = OpenAI(
        model=model,
        api_key=os.getenv("OPENAI_API_KEY"),
        api_base=os.getenv("OPENAI_API_BASE"),
        streaming=True,
    )
    
    async for chunk in llm.astream_complete(prompt):
        print(chunk.delta, end="", flush=True)
    print()

# Usage
asyncio.run(stream_response("Write a haiku about routing", "claude-3-5-sonnet"))

For agents and workflows, pass the streaming LLM directly:

from llama_index.agent.openai import OpenAIAgent
from llama_index.llms.openai import OpenAI

streaming_llm = OpenAI(
    model="gpt-5",
    api_key=os.getenv("OPENAI_API_KEY"),
    api_base=os.getenv("OPENAI_API_BASE"),
    streaming=True,
)

agent = OpenAIAgent.from_tools(
    tools=my_tools,
    llm=streaming_llm,
    verbose=True,
)

# Streams tokens as the agent reasons
async for chunk in agent.astream_chat("Analyze this codebase"):
    print(chunk.delta, end="")

Step 8: Deploy with confidence

Three operational concerns when you switch llm provider in llamaindex at scale:

Rate limits and quotas — The unified endpoint handles provider-level rate limits with automatic fallback. Configure your fallback chain in the x-n4n-fallback header or set a default in the dashboard. No client-side retry logic needed.

Latency variance — Different providers have different latency profiles. Log x-n4n-latency-ms from response headers and alert on p99 regressions:

import time
from llama_index.core.callbacks import BaseCallbackHandler

class LatencyLogger(BaseCallbackHandler):
    def on_llm_end(self, response, **kwargs):
        latency = response.raw.get("headers", {}).get("x-n4n-latency-ms")
        if latency:
            print(f"Provider latency: {latency}ms")

Settings.callback_manager.handlers.append(LatencyLogger())

Cost attribution — Tag requests with x-n4n-metadata for per-team, per-feature billing:

response = llm.chat(
    messages=[ChatMessage(role="user", content=query)],
    extra_headers={
        "x-n4n-metadata": "team=search,feature=rag,environment=prod",
    }
)

The usage API returns aggregated spend by these tags — no instrumentation changes required.

What you’ve built

In five minutes you now have:

  • A single endpoint serving 240+ models via OpenAI-compatible API
  • Runtime model selection without code deployments
  • Automatic fallback when providers degrade
  • Per-request routing directives for task-specific models
  • Standardized usage and latency observability
  • Streaming support for agents and chat interfaces

The pattern scales: add new models by updating the resolver, not rewriting pipelines. Your LlamaIndex code — indexes, agents, workflows, query engines — stays provider-agnostic.

Tagsllamaindexmulti-providerintegration

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Framework tutorialsSwapping models mid-conversation with LlamaIndexLearn to swap models mid-conversation in LlamaIndex with a conversation manager that preserves context, handles provider fallbacks, and routes dynamically across OpenAI, Anthropic, and local models.Framework tutorialsHow to avoid vendor lock-in with LangChain or LlamaIndexPractical steps to avoid llm vendor lock-in langchain llamaindex by routing both frameworks through one OpenAI-compatible gateway with code.Framework tutorialsMulti-model routing in AutoGen: GPT-5 and Llama 4Build a multi-model AutoGen workflow that routes between GPT-5 and Llama 4 via a single OpenAI-compatible endpoint, with fallback and per-token metering.Framework tutorialsProvider-agnostic agents: swapping models in CrewAI safelyLearn to swap LLM models in CrewAI without rewriting agents — configure provider-agnostic routing, handle model-specific quirks, and verify behavior across OpenAI, Anthropic, and local models.Framework tutorialsRunning the same LangGraph agent across four model providersLearn to run a LangGraph agent across GPT-5, Claude, Gemini, and Llama using a single OpenAI-compatible endpoint with automatic fallback and per-token metering.Framework tutorialsHow to swap GPT-5 for Claude Opus 4.5 in LangChainLearn how to swap GPT-5 for Claude Opus 4.5 in LangChain using native Anthropic SDK or an OpenAI-compatible gateway, with code and verification steps.

All one backend, every model: swapping gpt-5, claude, gemini & llama across frameworks posts →