Setting up llamaindex agent tool calling openai-compatible endpoints is straightforward if you treat the gateway as a drop-in OpenAI substitute. This tutorial builds a working agent that invokes custom Python functions, then shows how to point it at any OpenAI-compatible backend without rewriting your logic.
Prerequisites
- Python 3.10 or newer
llama-indexcore plus the OpenAI LLM integration- An API key and base URL for an OpenAI-compatible service (OpenAI, a self-hosted vLLM server, or a gateway)
Set these environment variables before running anything:
export OPENAI_API_KEY="sk-your-key"
export OPENAI_BASE_URL="https://api.openai.com/v1"If you use a gateway, swap the base URL. Your agent code will not care.
Install dependencies
pip install "llama-index" "llama-index-llms-openai>=0.1.0"That pulls the core agent runtime and the OpenAI-compatible LLM wrapper. No special tool-calling plugin is required; LlamaIndex serializes Python functions to OpenAI’s functions schema automatically.
Define tools as plain functions
LlamaIndex’s FunctionTool wraps a typed Python callable and generates the JSON schema the model sees. Keep docstrings precise—the model reads them to decide when to call.
from llama_index.tools import FunctionTool
def get_weather(city: str) -> str:
"""Return current weather summary for a given city."""
# Mock: replace with a real API call
return f"Sunny, 22C in {city}"
def multiply(a: float, b: float) -> float:
"""Multiply two numbers and return the product."""
return a * b
weather_tool = FunctionTool.from_defaults(fn=get_weather)
math_tool = FunctionTool.from_defaults(fn=multiply)The docstring becomes the tool description. Bad or vague docstrings produce flaky tool selection. Write them like API docs.
Configure the OpenAI-compatible LLM
The OpenAI class from llama_index.llms.openai accepts api_base. That is the only hook you need for llamaindex agent tool calling openai-compatible backends.
import os
from llama_index.llms.openai import OpenAI
llm = OpenAI(
model="gpt-3.5-turbo",
api_key=os.environ["OPENAI_API_KEY"],
api_base=os.environ["OPENAI_BASE_URL"],
temperature=0,
timeout=30,
)If your gateway uses a different model name (e.g., openai/gpt-4o or anthropic/claude-3), set model to that string. The LLM wrapper forwards it verbatim.
Build the agent
OpenAIAgent is the LlamaIndex runner that loops: model decides tools, tools execute, results return to the model until it produces a final answer.
from llama_index.agents.openai_agent import OpenAIAgent
agent = OpenAIAgent.from_tools(
tools=[weather_tool, math_tool],
llm=llm,
verbose=True,
max_function_calls=5,
)max_function_calls prevents runaway loops. Set it based on your latency budget.
Run a synchronous query
response = agent.chat("What is the weather in Tokyo and what is 12 * 9?")
print(str(response))With verbose=True you’ll see the agent’s internal steps:
Calling function get_weather with args: {"city": "Tokyo"}
Calling function multiply with args: {"a": 12, "b": 9}
Final response: Sunny, 22C in Tokyo. 12 * 9 = 108.The exact log format varies by version, but the tool arguments print before the synthesized answer. If you see the model ignore a tool, check that the model actually supports function calling—some older or distilled checkpoints do not.
Access tool call metadata
Production logging needs more than the printed string. The response object carries metadata:
print(response.metadata)
# {'tool_calls': [{'name': 'get_weather', 'args': {'city': 'Tokyo'}},
# {'name': 'multiply', 'args': {'a': 12, 'b': 9}}]}Pipe this to your tracing stack to reconstruct call graphs.
Async execution
Most serving stacks are async. Use achat to avoid blocking the event loop.
import asyncio
async def main():
agent = OpenAIAgent.from_tools([weather_tool, math_tool], llm=llm)
resp = await agent.achat("Weather in Paris and 7 * 6?")
print(str(resp))
asyncio.run(main())The async path uses the same tool wrappers; LlamaIndex runs synchronous functions in a threadpool.
Streaming intermediate output
For chat UIs, stream the agent’s final text:
from llama_index.llms.openai import OpenAI
streaming_llm = OpenAI(
model="gpt-3.5-turbo",
api_key=os.environ["OPENAI_API_KEY"],
api_base=os.environ["OPENAI_BASE_URL"],
stream=True,
)
streaming_agent = OpenAIAgent.from_tools([weather_tool], llm=streaming_llm)
stream = streaming_agent.stream_chat("Weather in Berlin?")
for token in stream.response_gen:
print(token, end="")Tool calls still happen non-streamingly; only the final natural-language response streams.
Error handling and timeouts
Tools fail. Wrap your function bodies so the agent gets a string error rather than an exception trace:
def get_weather(city: str) -> str:
"""Return current weather summary for a given city."""
try:
# real network call here
return f"Sunny, 22C in {city}"
except Exception as e:
return f"ERROR: {e}"The model can often recover from a returned error string by retrying with different arguments. Unhandled exceptions kill the agent loop.
Production routing and fallback
When you move past a single provider, point OPENAI_BASE_URL at a gateway that speaks the OpenAI protocol. For example, n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models and automatic fallback when a provider is rate-limited or degraded. Your LlamaIndex agent code stays exactly as written—only the environment variable changes. Because the gateway forwards provider cache-control hints and honors client routing directives, you can pin a model per request via the model field without custom middleware.
# Route to a specific provider model through the gateway
llm = OpenAI(
model="openai/gpt-4o-mini",
api_key=os.environ["OPENAI_API_KEY"],
api_base=os.environ["OPENAI_BASE_URL"],
)Common pitfalls
Model capability mismatch. Not every model behind an OpenAI-compatible endpoint implements tools. If tool calls silently vanish, curl the /models endpoint and verify the model card claims function calling.
Schema drift. Complex Pydantic types in function signatures sometimes serialize to schemas the model mishandles. Prefer flat arguments with primitive types.
Latency amplification. Each tool round-trip is a full model inference. A three-step plan on a 2-second endpoint costs six seconds plus tool runtime. Cache tool results inside the function if they are idempotent.
Verbose logging in prod. verbose=True prints arguments that may contain PII. Gate it behind a debug flag.
Verify the integration end-to-end
Write a tiny test that asserts the agent used the math tool:
def test_multiply_tool():
resp = agent.chat("Calculate 3 * 4")
assert "12" in str(resp)
assert any(t["name"] == "multiply" for t in resp.metadata["tool_calls"])
test_multiply_tool()Run it against your real base URL in CI with a cheap model. This catches regressions when you swap providers or upgrade LlamaIndex.
That is the full loop: define functions, wrap them, point the LLM at any OpenAI-compatible URL, and run. The llamaindex agent tool calling openai-compatible pattern keeps your application code stable while the backend shifts under you.