Most agents break the moment a single model can’t handle a tool call reliably. To route function calls across models without rewriting your agent loop, you need a thin dispatch layer that speaks one API shape and forwards tool schemas verbatim.
Step 1: Define a single function-calling contract
Function calling lives or dies on schema discipline. Write your tool definitions once as JSON Schema and reuse them for every provider that accepts the OpenAI chat completions format.
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Fetch current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "City, state, country"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
},
"required": ["location"]
}
}
}Keep descriptions terse but unambiguous. Models vary in how strictly they follow required; treat missing fields as a retry signal, not a crash.
In Python, load these schemas and pass them to any OpenAI-compatible client:
from openai import OpenAI
import json
client = OpenAI(base_url="https://api.openai.com/v1", api_key="YOUR_KEY")
tools = [json.load(open("get_weather.json"))]
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Weather in Berlin?"}],
tools=tools,
tool_choice="auto"
)Step 2: Map tasks to model capabilities
Not every model parses parallel tool calls or nested schemas equally. Build a static registry that lists which models handle which call patterns. You are not benchmarking; you are recording observed behavior from your own traffic.
MODEL_REGISTRY = {
"simple_tools": ["gpt-4o-mini", "claude-3-haiku", "mixtral-8x7b"],
"strict_schema": ["gpt-4o", "claude-3-5-sonnet"],
"parallel_calls": ["gpt-4o", "claude-3-5-sonnet", "gemini-1.5-pro"]
}Route function calls across models by selecting the cheapest entry that meets the call complexity. If the user request likely triggers one tool, use simple_tools. If your agent plans a multi-tool sequence, pick from parallel_calls. This registry is the only place that names concrete models; the rest of your code stays agnostic.
Step 3: Implement the router with explicit fallback
A router is a loop over candidate models. Try the primary; on exception or malformed tool call, shift to the next. A gateway like n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models and performs automatic fallback when a provider is rate-limited or degraded, so your code only handles explicit routing directives and logic errors.
def route_function_call(messages, tools, candidate_models):
last_err = None
for model in candidate_models:
try:
resp = client.chat.completions.create(
model=model,
messages=messages,
tools=tools,
tool_choice="auto"
)
msg = resp.choices[0].message
if not msg.tool_calls:
# Model ignored the tool; treat as soft failure
last_err = f"{model} returned no tool_calls"
continue
return model, resp
except Exception as e:
last_err = f"{model} raised {e}"
continue
raise RuntimeError(f"All candidates failed: {last_err}")Call it with your registry:
model, resp = route_function_call(
messages=[{"role": "user", "content": "Weather in Berlin?"}],
tools=tools,
candidate_models=MODEL_REGISTRY["simple_tools"]
)
print(f"Handled by {model}")The loop catches both hard errors (HTTP 5xx, timeout) and soft errors (no tool_calls when you expected them). That distinction matters: a soft failure means the model is alive but useless for this turn.
Step 4: Forward client routing directives and cache hints
Some gateways let you pin a provider or signal cache control. Use extra_body for routing and embed cache_control in tool parameter schemas where the upstream provider supports it. This keeps your intent explicit without custom headers.
resp = client.chat.completions.create(
model="claude-3-5-sonnet",
messages=messages,
tools=tools,
extra_body={
"routing": {"provider": "anthropic", "fallback": ["openai", "google"]},
"cache_control": {"type": "ephemeral"}
}
)If you define a tool that rarely changes, mark it cacheable to cut prompt tokens on repeated calls:
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Fetch current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string"}
},
"required": ["location"]
},
"cache_control": {"type": "ephemeral"}
}
}Honor the gateway’s forwarding of these hints; don’t reinvent cache logic in your agent. The point is to route function calls across models while letting the transport layer optimize what it can.
Step 5: Meter per-token usage
Per-token metering is non-negotiable when you route function calls across models because cost variance between a Haiku and a Sonnet call is significant. Capture usage on every response and tag it with the model that actually served the call.
import logging
def log_usage(model, usage):
logging.info({
"model": model,
"prompt_tokens": usage.prompt_tokens,
"completion_tokens": usage.completion_tokens,
"total_tokens": usage.total_tokens
})
model, resp = route_function_call(...)
log_usage(model, resp.usage)Pipe this to your metrics backend. When a fallback fires, you will see the secondary model’s higher token count; that’s the cost of resilience. Without per-token tags you cannot tell whether a routing rule is saving or burning money.
Step 6: Verify the pipeline end to end
You need proof the router works before production. Write a test that forces a failure on the first candidate and asserts the second handles the tool call.
def test_fallback(monkeypatch):
calls = {"n": 0}
def fake_create(*args, **kwargs):
calls["n"] += 1
if calls["n"] == 1:
raise RuntimeError("provider 503")
class Msg:
tool_calls = [{"id": "1", "function": {"name": "get_weather"}}]
class Choice: message = Msg()
class Usage: prompt_tokens = 10; completion_tokens = 5; total_tokens = 15
class R: choices = [Choice()]; usage = Usage()
return R()
monkeypatch.setattr(client.chat.completions, "create", fake_create)
model, resp = route_function_call(
messages=[], tools=tools,
candidate_models=["fail-model", "backup-model"]
)
assert model == "backup-model"For live verification, run the agent against a sandboxed tool that echoes its input. Confirm the correct tool_calls appear, the served model matches your registry, and usage logs populate. If you can trigger a provider outage in staging, watch the fallback engage without a 500 to the client. Success means: (1) the right tool name is emitted, (2) the model that served is logged, (3) a killed primary automatically yields a working secondary.
Common pitfalls when you route function calls across models
Schema drift is the silent killer. If you edit a tool definition for one model’s quirks, you fracture the contract. Keep one canonical schema and let the router add model-specific extra_body only.
Another trap: trusting tool_choice="required" on models that don’t support it. Degrade gracefully to auto and validate the response. If the response has no tool call, your router should treat it as a soft failure and move on.
Finally, don’t centralize model selection in a static file forever. As new models land, extend the registry and let real usage data promote or demote entries. A model that was strict last month may be lax after an update.
Closing checklist
- One JSON Schema per tool, reused everywhere.
- Registry maps task type to ordered model list.
- Router loops with soft and hard failure handling.
- Routing directives and cache hints passed via
extra_body. - Usage metered per model on every call.
- Integration test proves fallback works.
Follow these steps and you can route function calls across models without coupling your agent to any single vendor’s quirks.