LangChain agents with tool calling on OpenAI-compatible models

Building langchain agents tool calling openai-compatible models is mostly a configuration exercise: point ChatOpenAI at a compliant base URL and use the standard tool-calling agent primitive. This tutorial ships a runnable agent that queries custom tools, shows the wire-level behavior, and covers resilience patterns you’ll need before shipping to production.

Prerequisites

• Python 3.10 or newer
• langchain>=0.2 and langchain-openai>=0.1
• An OpenAI-compatible endpoint (self-hosted vLLM, llama.cpp, or a gateway)
• Environment variables OPENAI_API_KEY and OPENAI_API_BASE exported

pip install langchain langchain-openai python-dotenv

# load_env.py
import os
from dotenv import load_dotenv

load_dotenv()
assert os.environ.get("OPENAI_API_BASE")
assert os.environ.get("OPENAI_API_KEY")

You should know basic Python decorators. Async is optional but covered briefly.

Step 1: Configure the chat model

LangChain’s ChatOpenAI speaks the OpenAI chat completions protocol. Any endpoint that mirrors /v1/chat/completions and accepts tools works as a drop-in.

import os
from langchain_openai import ChatOpenAI

model = ChatOpenAI(
    model="gpt-4o-mini",
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ["OPENAI_API_BASE"],
    temperature=0,
    max_retries=2,
)

Avoid trailing slashes on OPENAI_API_BASE. The client appends /chat/completions internally; a double slash breaks some strict servers.

Step 2: Define tools with schema

The @tool decorator extracts the name, description, and argument schema from type hints and docstrings. This schema is what gets sent to the model.

from langchain.tools import tool

@tool
def get_weather(city: str) -> str:
    """Return current temperature for a city."""
    # Mock; replace with requests to a real API
    return f"22C in {city}"

@tool
def multiply(a: int, b: int) -> int:
    """Multiply two integers."""
    return a * b

tools = [get_weather, multiply]

# Inspect generated schema
print(get_weather.args)
# {'city': {'title': 'City', 'type': 'string'}}

Missing docstrings cause LangChain to silently drop the tool. Write precise descriptions; the model routes on them.

Step 3: Assemble the tool-calling agent

Use create_tool_calling_agent. It requires a prompt with a agent_scratchpad placeholder where intermediate tool calls render.

from langchain_core.prompts import ChatPromptTemplate
from langchain.agents import AgentExecutor, create_tool_calling_agent

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a precise assistant that uses tools when needed."),
    ("human", "{input}"),
    ("placeholder", "{agent_scratchpad}"),
])

agent = create_tool_calling_agent(model, tools, prompt)
executor = AgentExecutor(
    agent=agent,
    tools=tools,
    verbose=True,
    return_intermediate_steps=True,
)

Step 4: Run a multi-tool query

response = executor.invoke({
    "input": "What is 6 times 7 and the weather in Paris?"
})
print(response["output"])

Expected console output (abridged):

> Entering new AgentExecutor chain...
Invoking: `multiply` with `{'a': 6, 'b': 7}`
Invoking: `get_weather` with `{'city': 'Paris'}`

6 times 7 is 42. The weather in Paris is 22C.
> Finished chain.

The agent emitted two tool calls in a single reasoning step, then synthesized the final string. That’s the core loop for langchain agents tool calling openai-compatible deployments.

Step 5: Inspect token usage

OpenAI-compatible responses embed a usage object. LangChain surfaces it as usage_metadata on the message.

raw = model.invoke("Say hi")
print(raw.usage_metadata)
# {'input_tokens': 9, 'output_tokens': 2, 'total_tokens': 11}

When you route through a gateway, per-token metering arrives in the same shape, so existing logging and cost dashboards work unchanged.

Full runnable script

import os
from langchain_openai import ChatOpenAI
from langchain.tools import tool
from langchain_core.prompts import ChatPromptTemplate
from langchain.agents import AgentExecutor, create_tool_calling_agent

@tool
def get_weather(city: str) -> str:
    """Return current temperature for a city."""
    return f"22C in {city}"

@tool
def multiply(a: int, b: int) -> int:
    """Multiply two integers."""
    return a * b

tools = [get_weather, multiply]

model = ChatOpenAI(
    model="gpt-4o-mini",
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ["OPENAI_API_BASE"],
    temperature=0,
)

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a precise assistant that uses tools when needed."),
    ("human", "{input}"),
    ("placeholder", "{agent_scratchpad}"),
])

agent = create_tool_calling_agent(model, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, verbose=False)

if __name__ == "__main__":
    out = executor.invoke({"input": "What is 6 times 7 and the weather in Paris?"})
    print(out["output"])

Run with python script.py. You should see 6 times 7 is 42. The weather in Paris is 22C.

Resilience: retries and fallback

Single-endpoint setups fail on downtime. The simplest fix is client retries:

model = ChatOpenAI(
    ...,
    max_retries=3,
    request_timeout=10,
)

For multi-provider routing, you don’t want agent code littered with fallback branches. A gateway that honors client routing directives and automatically fails over when a provider is rate-limited or degraded keeps the agent logic clean. For example, n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models with that behavior, so the script above runs unchanged against shifting backend availability.

Streaming with tool calls

Set streaming=True to get token events, but note the agent executor buffers tool calls until the step completes. You can observe chain events via astream_events:

import asyncio
from langchain_openai import ChatOpenAI
from langchain.agents import AgentExecutor, create_tool_calling_agent

async def main():
    stream_model = ChatOpenAI(..., streaming=True)
    agent = create_tool_calling_agent(stream_model, tools, prompt)
    exec = AgentExecutor(agent=agent, tools=tools)
    async for ev in exec.astream_events({"input": "Weather in Tokyo?"}):
        if ev["event"] == "on_chain_end":
            print(ev["data"]["output"])

asyncio.run(main())

Tool arguments are not streamed token-by-token; the call is atomic once the model commits to it.

Testing the agent

Pin a fake model or use langchain_core stubs in unit tests. A minimal pytest checks tool wiring without network:

def test_tools_registered():
    assert any(t.name == "multiply" for t in tools)
    assert multiply.invoke({"a": 3, "b": 4}) == 12

For integration tests, set OPENAI_API_BASE to a local mock that echoes tool calls.

Common pitfalls

• Missing docstring: Tool omitted from schema.
• Unsupported model: Not every OpenAI-compatible weight implements tools. Validate with gpt-4o-mini first.
• Wrong placeholder: Prompt must include ("placeholder", "{agent_scratchpad}") or the agent crashes.
• Timeout storms: Default request_timeout is 60s; lower it in latency-sensitive paths.

Extending the pattern

Swap create_tool_calling_agent for a custom Runnable if you need human-in-the-loop approval. Add a RunnableLambda to sanitize tool outputs. The invariant remains: as long as the endpoint speaks the OpenAI chat protocol with tools, your langchain agents tool calling openai-compatible code stays portable across providers and self-hosted stacks.

You now have a runnable agent, usage visibility, and a fallback story. Ship it behind a queue and you’re production-ready.