Implementing llamaindex gpt-4o llama 3.1 routing lets you serve high-complexity queries with GPT-4o and offload bulk or latency-sensitive traffic to Llama 3.1 without maintaining two code paths. This guide shows how to stand up both models behind LlamaIndex’s OpenAI-compatible interfaces and switch between them per request using explicit routing logic.
Step 1: Install dependencies and import modules
LlamaIndex splits LLM providers into separate packages. For OpenAI and any OpenAI-compatible endpoint you only need llama-index-llms-openai.
pip install llama-index-core llama-index-llms-openai python-dotenvImport the pieces you will actually use:
import os
from dotenv import load_dotenv
from llama_index.core import Settings, VectorStoreIndex, Document
from llama_index.llms.openai import OpenAI, OpenAILike
load_dotenv()Keep your dependency surface small. If you are not using the full llama-index meta-package, you avoid pulling in dozens of unused integrations.
Step 2: Configure the two model clients
GPT-4o goes through OpenAI’s official API. Llama 3.1 is an open-weight model served by many providers; the cleanest way to avoid per-provider SDK churn is to treat it as an OpenAI-compatible endpoint. If you want a single endpoint that addresses both proprietary and open-weight models, point OpenAILike at a gateway such as n4n.ai, which exposes one OpenAI-compatible API across 240+ models and honors client routing directives.
# GPT-4o via OpenAI
llm_gpt4o = OpenAI(
model="gpt-4o",
api_key=os.getenv("OPENAI_API_KEY"),
temperature=0.0,
)
# Llama 3.1 70B via an OpenAI-compatible gateway
llm_llama31 = OpenAILike(
model="meta-llama/llama-3.1-70b-instruct",
api_base=os.getenv("OPENAI_COMPAT_BASE"), # e.g. https://api.n4n.ai/v1
api_key=os.getenv("OPENAI_COMPAT_KEY"),
temperature=0.0,
)Store keys in .env:
OPENAI_API_KEY=sk-...
OPENAI_COMPAT_BASE=https://api.n4n.ai/v1
OPENAI_COMPAT_KEY=sk-...The core of llamaindex gpt-4o llama 3.1 routing is a unified interface: both objects expose .chat(), .complete(), and the same metadata shape, so your calling code stays identical.
Step 3: Define an explicit routing policy
Do not hide model selection inside a vague “auto” router. Write a small, testable function that maps query characteristics to a model. A useful heuristic: route to GPT-4o when the query implies multi-step reasoning, code generation, or low tolerance for hallucination; send everything else to Llama 3.1.
class ModelRouter:
def __init__(self, gpt4o, llama31):
self.gpt4o = gpt4o
self.llama31 = llama31
self._complexity_markers = [
"why", "explain", "debug", "refactor",
"compare", "design", "prove"
]
def select(self, query: str):
q = query.lower()
if any(marker in q for marker in self._complexity_markers):
return self.gpt4o
if len(query.split()) > 120:
return self.gpt4o
return self.llama31
router = ModelRouter(llm_gpt4o, llm_llama31)In production you might replace the keyword check with a cheap classifier or a latency budget check. The point is that the routing decision is visible and logged.
Step 4: Wire routing into a LlamaIndex query flow
LlamaIndex uses a global Settings.llm for index query engines. Swap it inside your request handler. For a runnable example, build an in-memory index from a single document:
doc = Document(text="""
LlamaIndex is a data framework for LLM applications.
It connects custom data sources to large language models.
GPT-4o is OpenAI's multimodal flagship model.
Llama 3.1 is Meta's open-weight model family.
""")
index = VectorStoreIndex.from_documents([doc])Now write the routed query function:
def routed_query(query: str) -> str:
llm = router.select(query)
Settings.llm = llm
engine = index.as_query_engine(similarity_top_k=1)
response = engine.query(query)
# attach model id for verification
model_id = llm.metadata.model_name
print(f"[routed to {model_id}]")
return str(response)Call it:
print(routed_query("What is LlamaIndex?"))
print(routed_query("Explain why LlamaIndex improves RAG latency compared to raw API calls."))The first query hits Llama 3.1; the second contains “explain” and routes to GPT-4o. This pattern keeps llamaindex gpt-4o llama 3.1 routing explicit and testable without subclassing query engines.
If you need concurrent requests with different models, do not mutate global Settings from multiple threads. Instead pass the llm directly to a RetrieverQueryEngine instance:
from llama_index.core.query_engine import RetrieverQueryEngine
def routed_query_concurrent(query: str) -> str:
llm = router.select(query)
retriever = index.as_retriever(similarity_top_k=1)
engine = RetrieverQueryEngine.from_args(retriever, llm=llm)
return str(engine.query(query))Step 5: Verify the routing works
Verification is two-fold: confirm the expected model answered, and confirm the responses are coherent.
Add a tiny assertion helper:
def assert_routing(query, expected_model_substr):
llm = router.select(query)
assert expected_model_substr in llm.metadata.model_name
print(f"OK: '{query[:30]}...' -> {llm.metadata.model_name}")
assert_routing("List the models mentioned.", "llama")
assert_routing("Compare GPT-4o and Llama 3.1 tradeoffs.", "gpt-4o")Run the script. You should see:
OK: 'List the models mention...' -> meta-llama/llama-3.1-70b-instruct
OK: 'Compare GPT-4o and Llam...' -> gpt-4o
[routed to meta-llama/llama-3.1-70b-instruct]
LlamaIndex is a data framework...
[routed to gpt-4o]
GPT-4o and Llama 3.1 differ in...If the assertions pass and the printed model IDs match the heuristic, your llamaindex gpt-4o llama 3.1 routing is functioning. For production, emit the model ID to your metrics pipeline alongside per-token usage so you can audit cost savings.
Edge cases worth handling
- Provider degradation: When GPT-4o is rate-limited, fall back to Llama 3.1 instead of erroring. Wrap
engine.queryin try/except and re-route onRateLimitError. - Streaming: Both
OpenAIandOpenAILikesupportstream_complete. Your router should stay identical; just callllm.stream_complete()and iterate. - Cache hints: If your gateway forwards provider cache-control hints, set
extra_headers={"cache-control": "max-age=300"}on the Llama 3.1 client to cut repeat-query cost.
Why not use RouterQueryEngine?
RouterQueryEngine is designed to pick between multiple knowledge sources (e.g., a SQL index vs. a vector index), not between LLMs. Shoving model selection into it forces you to build dummy tools and adds indirection. A plain router object and a temporary Settings.llm swap is 20 lines and debuggable in an hour.
Production notes
Once this pattern is proven, move the router behind your API boundary. Accept a prefer_model field in the request schema to let callers override the heuristic. Keep the default policy in code, not config, so you can unit-test it. If you adopt a gateway that honors client routing directives, you can also push the selection server-side, but client-side routing gives you stack traces when the logic breaks.
Verifying llamaindex gpt-4o llama 3.1 routing in CI is cheap: mock the LLM clients, assert select() returns the right object, and you are covered. The runtime swap is boring on purpose.