Most LlamaIndex examples assume OpenAI’s hosted API, but production systems rarely live there. The llamaindex query engine base_url config lets you point the engine at any OpenAI-compatible server—a local vLLM instance, a corporate proxy, or an inference gateway—without rewriting your retrieval logic. This article walks through the exact code paths and configuration objects you need to swap the endpoint and verify it works.
Step 1: Install and pin the LlamaIndex stack
Use a clean virtual environment. LlamaIndex ships frequent breaking changes, so pin major versions.
pip install "llama-index==0.10.*" "openai==1.*"
python -c "import llama_index; print(llama_index.__version__)"If the import fails or the version prints older than 0.10.0, upgrade before continuing. The base_url parameter on the LLM wrapper has been stable since 0.9.20, but the Settings object is the current recommended pattern.
Step 2: Instantiate the LLM with a custom base_url
LlamaIndex’s OpenAI class accepts base_url directly. Point it at your endpoint’s /v1 path if the server expects it.
from llama_index.llms.openai import OpenAI
llm = OpenAI(
model="mistral-7b-instruct",
api_key="not-a-real-key",
base_url="http://localhost:8000/v1",
temperature=0.1,
timeout=30,
)
print(llm.base_url) # http://localhost:8000/v1If you route through an OpenAI-compatible gateway like n4n.ai, its single endpoint fronts 240+ models and performs automatic fallback when a provider is degraded—set base_url to that endpoint and pass the model name as the route.
Trailing slashes cause 404s on some servers. Strip them unless the server docs explicitly require one.
Step 3: Register the LLM in LlamaIndex Settings
The query engine pulls its LLM from Settings (or a ServiceContext in legacy code). Set it globally so every index inherits the endpoint.
from llama_index import Settings
Settings.llm = llm
# Optional: if your embedding model is also custom, set it here (see Step 8)
# Settings.embed_model = custom_embedFor older codebases using ServiceContext:
from llama_index import ServiceContext
service_context = ServiceContext.from_defaults(llm=llm)Both approaches achieve the same llamaindex query engine base_url config binding; Settings is less boilerplate.
Step 4: Build an index and derive a query engine
Load documents and create a VectorStoreIndex. The query engine inherits the LLM we just configured.
from llama_index import VectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()No base_url appears in this snippet because the llamaindex query engine base_url config is already attached via Settings.llm. If you instantiated the index with a service_context, pass service_context=service_context to from_documents.
Step 5: Run a synchronous query
Execute a query and print the response. This forces a round-trip to your custom endpoint.
response = query_engine.query("What is the refund policy in the docs?")
print(str(response))If you see a valid answer, the request reached the server. If you get ConnectionRefusedError, the base_url host is wrong or the server is down.
Step 6: Verify the request hit your custom endpoint
Don’t trust the client alone. Confirm server-side.
Option A: Server logs. A local vLLM or LiteLLM instance prints incoming requests with the model name and path. Look for POST /v1/chat/completions.
Option B: Transparent proxy. Run a logging proxy:
pip install mitmproxy
mitmproxy --listen-port 8080 --mode reverse:http://localhost:8000Then set base_url="http://localhost:8080/v1". The proxy shows the exact payload.
Option C: Independent curl. Validate the endpoint contract outside LlamaIndex:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"mistral-7b-instruct","messages":[{"role":"user","content":"ping"}]}'If curl works but LlamaIndex fails, the issue is in the request shape LlamaIndex sends (often temperature or max_tokens defaults).
Step 7: Enable streaming and pass extra parameters
Production query engines should stream to avoid blocking on long generations. The llamaindex query engine base_url config works identically for streaming.
streaming_llm = OpenAI(
model="mistral-7b-instruct",
api_key="not-a-real-key",
base_url="http://localhost:8000/v1",
streaming=True,
)
Settings.llm = streaming_llm
streaming_query_engine = index.as_query_engine(streaming=True)
streaming_response = streaming_query_engine.query("Summarize the appendix.")
for token in streaming_response.response_gen:
print(token, end="")Additional kwargs (max_tokens, top_p) go into the OpenAI constructor. They are forwarded as-is to the endpoint.
Step 8: Configure the embedding model base_url
If your vector index uses a custom embedding server, set it separately. The query engine uses embeddings during index build and (for some retrievers) during query expansion.
from llama_index.embeddings.openai import OpenAIEmbedding
embed_model = OpenAIEmbedding(
model="text-embedding-ada-002",
api_key="not-a-real-key",
base_url="http://localhost:8001/v1",
)
Settings.embed_model = embed_modelRebuild the index after changing embed_model; existing indices store vectors from the old endpoint and will mismatch.
Troubleshooting and gotchas
404 on /v1/chat/completions. Your server may not prefix with /v1. Set base_url="http://host:port" and let LlamaIndex add the path.
Authentication errors with a fake key. Some gateways require a real key even for local routing. Use api_key="sk-placeholder" if the server ignores it, but check the server’s auth middleware.
Model not found. The model string must match what the endpoint serves. LlamaIndex does not validate this; it passes the string through.
Timeout under load. Raise the timeout in the OpenAI constructor. Default is 60s; embeddings and large contexts need more.
Verify success checklist
-
print(llm.base_url)matches your target host. -
query_engine.query()returns a non-empty string without connection errors. - Server access logs show a request from the LlamaIndex process.
- Streaming variant yields tokens incrementally.
- Embedding endpoint (if used) returns vectors of expected dimension.
The llamaindex query engine base_url config is now fully redirected. You can swap providers by changing one string, which is exactly what you want when a provider rate-limits or a self-hosted node goes offline.