Most agent failures trace back to the model calling the wrong function, not to bad reasoning. Testing tool selection accuracy in LLM agents requires more than a happy-path smoke test; you need a repeatable harness that asserts which tool was invoked and with what arguments.
Step 1: Define a fixed tool schema and a deterministic oracle
Lock your tool definitions in code so the schema cannot drift between the agent and the eval. The oracle is a pure function that maps a user prompt to the expected tool name and normalized arguments. It encodes the contract your tests enforce.
# tools.py
WEATHER_TOOL = {
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a location",
"parameters": {
"type": "object",
"properties": {"location": {"type": "string"}},
"required": ["location"]
}
}
}
CALC_TOOL = {
"type": "function",
"function": {
"name": "calculator",
"description": "Evaluate a math expression",
"parameters": {
"type": "object",
"properties": {"expression": {"type": "string"}},
"required": ["expression"]
}
}
}
TOOLS = [WEATHER_TOOL, CALC_TOOL]
def oracle(prompt: str) -> tuple[str, dict]:
"""Return (expected_tool, expected_args) for a given prompt."""
p = prompt.lower()
if "weather" in p or "temperature" in p:
loc = "San Francisco" if "sf" in p else "New York"
return "get_weather", {"location": loc}
if any(op in p for op in ["+", "-", "*", "/", "compute"]):
return "calculator", {"expression": "2+2"}
return "none", {}The oracle does not need to cover every edge case. It defines the behavior you refuse to break. Keep it side-effect free and version-controlled.
Step 2: Capture the model’s tool call without executing side effects
Never let the agent hit a real API or database during eval. Strip the execution layer and return the raw tool_calls object. Use an OpenAI-compatible client so the pattern transfers to any gateway.
# agent.py
from openai import OpenAI
import os
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
def select_tool(prompt: str, model: str = "gpt-4o-mini") -> dict | None:
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
tools=TOOLS,
tool_choice="auto",
)
msg = resp.choices[0].message
if not msg.tool_calls:
return None
tc = msg.tool_calls[0]
return {"name": tc.function.name, "arguments": tc.function.arguments}select_tool returns the invoked tool and a JSON string of arguments. Parse it with json.loads in the test. If the model returns no tool call, the function returns None—that is a valid outcome when the oracle expects "none".
Step 3: Write assertion-based eval cases
Use pytest. Assert on both the tool name and the parsed arguments. A test passes only when the model picks the right tool and supplies coherent slots.
# test_selection.py
import json
import pytest
from agent import select_tool
from tools import oracle
@pytest.mark.parametrize("prompt", [
"What's the weather in SF?",
"Will it rain in New York today?",
"Temperature in San Francisco?",
])
def test_weather_selection(prompt):
got = select_tool(prompt)
expected_name, expected_args = oracle(prompt)
assert got is not None, "No tool called"
assert got["name"] == expected_name
args = json.loads(got["arguments"])
assert args["location"] == expected_args["location"]
def test_calculator_selection():
got = select_tool("Compute 2+2")
assert got is not None
assert got["name"] == "calculator"
args = json.loads(got["arguments"])
assert "expression" in args
def test_no_tool_for_small_talk():
got = select_tool("Hey, how are you?")
assert got is None or got["name"] == "none"If you only check got["name"] == "get_weather", you miss cases where the model emits {"location": "SF"} instead of "San Francisco". Argument validation is part of testing tool selection accuracy in LLM agents.
Step 4: Run batch evaluations with seeded prompts
A single prompt is an anecdote. Build a JSONL dataset of 50–200 prompts with oracle labels, then run a batch loop that records pass/fail.
{"prompt": "weather in sf?"}
{"prompt": "what is 3*4?"}
{"prompt": "tell me a joke"}# run_eval.py
import json
from agent import select_tool
from tools import oracle
def evaluate(dataset_path: str):
results = []
with open(dataset_path) as f:
for line in f:
row = json.loads(line)
prompt = row["prompt"]
exp_name, exp_args = oracle(prompt)
got = select_tool(prompt)
ok = False
if exp_name == "none":
ok = got is None or got["name"] == "none"
elif got:
ok = got["name"] == exp_name
if ok and exp_args:
try:
args = json.loads(got["arguments"])
ok = all(args.get(k) == v for k, v in exp_args.items())
except Exception:
ok = False
results.append({"prompt": prompt, "pass": ok, "got": got})
return results
if __name__ == "__main__":
res = evaluate("eval_prompts.jsonl")
passes = sum(r["pass"] for r in res)
print(f"Accuracy: {passes}/{len(res)} = {passes/len(res):.2%}")Seed the dataset with adversarial variants: “weather in sf?” vs “sf weather?” vs “temperature in San Francisco”. The model’s tokenizer shouldn’t change your routing logic.
Step 5: Measure selection accuracy and argument correctness separately
Report two metrics. Tool selection rate is the fraction of cases where got["name"] == expected_name. Argument fidelity is the fraction where the parsed args match the oracle’s normalized values.
def metrics(results):
sel = sum(r["pass"] for r in results) / len(results)
return {"selection_accuracy": sel}When selection accuracy is 98% but argument fidelity is 80%, the agent is calling the right function and filling slots wrong—a different bug class. Testing tool selection accuracy in LLM agents without splitting these two numbers hides regressions. Add a confusion matrix if you have more than two tools; it shows whether get_weather is being confused with calculator or with no-op.
Step 6: Test fallback and routing behavior across models
Models differ in instruction adherence. Run the same eval suite against multiple model IDs to find which ones respect your schema. If you route through n4n.ai, you can pin a specific model per eval run via the provider routing header, exercising 240+ models from one OpenAI-compatible endpoint while keeping per-token metering for cost tracking.
# using n4n.ai routing directive (illustrative header)
client = OpenAI(
base_url="https://api.n4n.ai/v1",
api_key=os.environ["N4N_KEY"],
default_headers={"X-Route-Model": "anthropic/claude-3.5-sonnet"}
)Automatic fallback when a provider is degraded means your eval harness keeps running instead of throwing 429s. That’s infrastructure, not a test change. Honor client cache-control hints in your gateway so repeated eval prompts don’t burn tokens unnecessarily.
Verify success
Your eval is successful when:
pytestpasses on the parameterized unit tests for known prompts.- The batch script prints a selection accuracy number you can track in CI (e.g., gate on >= 95%).
- Argument fidelity stays above your threshold across at least three model IDs.
- Swapping the model name in
select_tooldoes not require changing the oracle or test logic. - Negative tests confirm the agent does not call a tool on small talk.
If those hold, you have a real signal on whether the agent picked the right tool—not a demo that worked once. Testing tool selection accuracy in LLM agents is fundamentally about deterministic contracts around nondeterministic models. Lock the schema, mock the side effects, assert on the call, and measure both axes. Do that and your agent stops guessing.