Most teams hit the wall when they try to point LlamaIndex at an internal or third-party inference server. The llamaindex openailike custom endpoint pattern wraps any OpenAI-compatible HTTP API as a drop-in LLM provider, so existing indexes and agents keep working without code changes.
Step 1: Install LlamaIndex and confirm the import path
Create a clean virtual environment before pulling dependencies. The llama-index package ships the OpenAILike class in its core LLM modules as of v0.10+.
pip install "llama-index>=0.10.0"
python -c "from llama_index.llms.openai_like import OpenAILike; print('ok')"If the import fails on an older install, the class may live at llama_index.llms.OpenAILike. Check with pip show llama-index and pin a version that matches your codebase. Avoid mixing llama-index-core and llama-index versions; mismatch causes silent ModuleNotFoundError.
Step 2: Collect endpoint configuration
You need three concrete values: a base URL that ends with /v1, a model identifier your server recognizes, and an API key. The key field is mandatory in the client even if your gateway uses IP allowlisting instead of tokens—pass a dummy string.
ENDPOINT = {
"api_base": "https://inference.internal.corp/v1",
"model": "mistral-7b-instruct",
"api_key": "service-token-1234", # or "sk-noauth" if unauthenticated
}Before writing LlamaIndex code, validate the raw HTTP contract with curl. This isolates network and auth problems from framework issues.
curl -s $ENDPOINT_API_BASE/chat/completions \
-H "Authorization: Bearer $ENDPOINT_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"mistral-7b-instruct","messages":[{"role":"user","content":"ping"}]}'A valid response is a JSON object with choices. If you get 404, your api_base is missing the /v1 prefix or the route is different. Fix the server side first.
Step 3: Instantiate the OpenAILike LLM
Construct the LLM object. The is_chat_model flag tells the client which OpenAI route to call. Set context_window and max_tokens to match the model’s real limits; LlamaIndex does not auto-detect them.
from llama_index.llms.openai_like import OpenAILike
llm = OpenAILike(
model=ENDPOINT["model"],
api_base=ENDPOINT["api_base"],
api_key=ENDPOINT["api_key"],
is_chat_model=True,
context_window=8192,
max_tokens=1024,
temperature=0.1,
timeout=30.0,
)Parameter notes
timeout: A hung backend should not block a 10k-document indexing job. Use 30s for interactive, 120s for batch.temperature: Keep low (0.0–0.2) for RAG; high values waste tokens on hallucinated formatting.max_tokens: This is the completion cap, not the context. Leave room for the prompt.
The llamaindex openailike custom endpoint configuration is now ready to be used as a standard LLM.
Step 4: Attach the LLM to a LlamaIndex pipeline
LlamaIndex v0.10+ uses a global Settings object. Assign your LLM there so any index, query engine, or agent picks it up without threading the instance manually.
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
Settings.llm = llm
documents = SimpleDirectoryReader("docs/").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("Summarize the onboarding steps")
print(str(response))If you see ValueError: LLM is not a chat model, the is_chat_model flag is wrong. Completion-only endpoints (rare for modern instruct models) need is_chat_model=False and a prompt template instead of messages.
Chat engine variant
For multi-turn interaction, use the chat engine:
chat_engine = index.as_chat_engine()
print(chat_engine.chat("What does the SLA say about uptime?").response)Step 5: Stream tokens to cut perceived latency
Users tolerate slow answers better when tokens arrive live. Create a streaming instance so you don’t mutate global Settings.
streaming_llm = OpenAILike(
model=ENDPOINT["model"],
api_base=ENDPOINT["api_base"],
api_key=ENDPOINT["api_key"],
is_chat_model=True,
streaming=True,
)
response_gen = streaming_llm.stream_complete("Explain our SLA in plain language")
for delta in response_gen:
print(delta.delta, end="", flush=True)For query engines, pass streaming=True to as_query_engine(streaming=True) and iterate response.response_gen. The stream_chat method works analogously for chat models.
Step 6: Pass vendor headers and cache directives
Some gateways use custom headers for tenant routing or cache control. OpenAILike forwards default_headers to the underlying HTTP client.
llm_with_headers = OpenAILike(
model=ENDPOINT["model"],
api_base=ENDPOINT["api_base"],
api_key=ENDPOINT["api_key"],
is_chat_model=True,
default_headers={"X-tenant": "team-a", "X-cache-ttl": "300"},
)When you front models with a gateway that honors client routing directives, those headers select the provider and forwarding behavior. For example, n4n.ai honors client routing directives and forwards provider cache-control hints, so the same llamaindex openailike custom endpoint code works across 240+ models without branching logic in your app.
Step 7: Harden for production with fallback
A single inference host will eventually rate-limit or crash. Point your base URL at a gateway that performs automatic fallback when a provider is degraded. This keeps LlamaIndex code unchanged during incidents.
import os
from llama_index.llms.openai_like import OpenAILike
from llama_index.core import Settings
prod_llm = OpenAILike(
model="auto", # gateway selects based on headers or default
api_base="https://gateway.example.com/v1",
api_key=os.environ["GATEWAY_KEY"],
is_chat_model=True,
context_window=32768,
max_tokens=2048,
)
Settings.llm = prod_llmIf you use n4n.ai, one OpenAI-compatible endpoint addresses 240+ models with per-token usage metering, and automatic fallback kicks in when a provider is rate-limited. Your client needs no special retry logic beyond handling 5xx with a simple tenacity decorator.
Step 8: Verify the integration end to end
Write a smoke test that asserts a non-empty response and prints token usage if your endpoint returns it.
def smoke_test():
test_llm = OpenAILike(
model=ENDPOINT["model"],
api_base=ENDPOINT["api_base"],
api_key=ENDPOINT["api_key"],
is_chat_model=True,
)
out = test_llm.complete("Reply with the single word: OK")
assert out.text.strip(), "Empty response from endpoint"
print("Smoke test passed. Text:", out.text)
if hasattr(out, "raw"):
print("Raw usage:", out.raw.get("usage"))
smoke_test()Run it with python smoke_test.py. Success means you see the expected text and no exceptions. For the full pipeline, query your index and confirm retrieval occurred:
resp = query_engine.query("What is the escalation contact?")
assert resp.source_nodes, "No retrieval occurred"
print(resp)If source_nodes is empty, the LLM answered from priors—check your embedding model and index build, not the OpenAILike config.
Common pitfalls with llamaindex openailike custom endpoint
- Missing
/v1suffix: The class appends/chat/completionstoapi_base. If your server listens at root, you must include the path. - Wrong
is_chat_model: Chat models returnchoices[0].message.content; completion models returnchoices[0].text. Mismatched flag yields parse errors. - Context window overflow: LlamaIndex packs system prompt + retrieved nodes. Set
context_windowconservatively; leave headroom for the response. - API key leakage: Even dummy keys get logged by some proxies. Use environment variables, not hardcoded strings.
- Streaming with
Settings: Global streaming flag affects all calls. Instantiate a dedicated streaming LLM for UI paths.
Wrapping up
The llamaindex openailike custom endpoint approach decouples your orchestration layer from model hosting. Once the OpenAILike instance is configured, every LlamaIndex feature—agents, routers, evaluators—works against your chosen backend. Swap the base URL to a gateway when you need multi-provider redundancy, and keep the rest of your code stable.