Detecting hallucinated tool calls AI agents emit is now a first-class QA problem, not a curiosity. When a model invents a function name or fills parameters with plausible-but-fake data, the failure surfaces only at runtime, often after a destructive side effect. The fix is to treat tool calls as untrusted input and verify them with the same rigor you apply to user form submissions.
1. Define a strict schema contract for every tool
A tool is only as safe as its schema. If you let the model call send_email(recipient, body) with free-form strings, you will get hallucinated addresses and fabricated names. Write the contract once, in JSON Schema, and generate the model-facing function definition from it. Avoid type: string where an enumeration or regex is knowable.
{
"name": "send_email",
"description": "Send an email to a verified internal user",
"parameters": {
"type": "object",
"properties": {
"recipient": {
"type": "string",
"pattern": "^[a-z0-9._]+@corp\\.example\\.com$"
},
"body": { "type": "string", "maxLength": 1000 }
},
"required": ["recipient", "body"]
}
}The pattern constraint rejects ceo@external.com or null. Loosely typed parameters are the most common source of silent hallucination. If you use Python, back the schema with a Pydantic model and generate both the OpenAI tool spec and the validator from the same source of truth:
from pydantic import BaseModel, constr
class SendEmailArgs(BaseModel):
recipient: constr(regex=r"^[a-z0-9._]+@corp\.example\.com$")
body: constr(max_length=1000)This eliminates drift between what the model is told and what your code enforces.
2. Validate at the boundary, not inside business logic
Never pass raw model output to your tool implementation. Wrap the agent loop with a validator that runs before any side effect. The validator is the only place that decides whether a call is legal.
import jsonschema
REGISTRY = {"send_email": send_email_impl}
def guard_tool_call(tool_name: str, args: dict, schema_map: dict):
if tool_name not in REGISTRY:
raise ValueError(f"unknown tool: {tool_name}")
jsonschema.validate(instance=args, schema=schema_map[tool_name])
return REGISTRY[tool_name](**args)If the model returns {"tool": "send_emaill", "args": {...}}, the tool_name not in REGISTRY check fails immediately. This is detecting hallucinated tool calls AI agents attempt without waiting for the SMTP server to bounce. The boundary guard also catches missing required fields and type mismatches.
Pitfall: returning a “corrected” call automatically. Correcting send_emaill to send_email trains your system to tolerate sloppy model output and hides regressions. Reject and log. The agent can retry if you return a structured error, but your code should never silently rewrite the invocation.
3. Build a fixed replay corpus
You need a set of transcripts where the model either called tools correctly or hallucinated. Capture production traces (with PII redacted) and augment with adversarial prompts designed to provoke invention:
- “Email the board about Q3” (no board address provided)
- “Delete the user with id 999999” (nonexistent id)
- “Restart the server in the us-east-1x region” (typo in region)
Store these as fixtures. Run them through the agent in a sandbox where tools are mocked.
@pytest.fixture
def corpus():
return load_jsonl("tests/fixtures/tool_calls.jsonl")Tradeoff: corpus maintenance is ongoing. Schemas change, new tools appear, and prompts that triggered hallucinations last month may be handled cleanly after a model update. Treat the corpus as code and review additions in PRs. A stale corpus gives false confidence.
4. Simulate with mocked side effects
Mock the tool registry so no real email sends or servers restart. Assert that invalid calls never reach the mock. The test should fail if a hallucinated call slips through the guard.
def test_hallucinated_tool_rejected(monkeypatch, corpus):
sent = []
monkeypatch.setattr("agent.REGISTRY", {"send_email": lambda **k: sent.append(k)})
for trace in corpus:
try:
run_agent(trace["prompt"])
except ValueError as e:
assert "unknown tool" in str(e) or "validation" in str(e)
# No invalid call should have executed
assert all(is_valid_record(t) for t in sent)This catches the class of bugs where the model fabricates a tool name or supplies an out-of-pattern argument. It does not catch semantic hallucinations (correct schema, wrong user id) — that requires deeper checks such as existence pre-flights in the mock.
5. Run model-in-the-loop regression tests
Schema validation is necessary but not sufficient. You must periodically run the real model to see if a new checkpoint regresses. Pin the model version to keep tests deterministic.
When executing these suites against multiple providers through a single OpenAI-compatible endpoint, n4n.ai lets you send a routing directive header to force a specific model build, so your regression run does not drift across provider updates. If a provider is degraded, its automatic fallback keeps the suite running, but you should pin the primary model for assertion stability.
curl https://api.n4n.ai/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-H "x-n4n-model: openai/gpt-4o-2024-08-06" \
-d '{ "model": "openai/gpt-4o-2024-08-06", "tools": [...], "messages": [...] }'In Python, use pytest with vcrpy to record and replay model responses, so you are not billed for every run but still test the parsing path. Record a cassette on a known model version, then assert that your validator rejects the hallucinated calls present in the recording. Nightly, re-record against the live pinned model to catch drift.
6. Runtime guards: reject, don’t silently correct
In production, wrap the same validator from step 2 in your serving layer. If a call fails validation, return a structured error to the agent and let it retry. After N retries, escalate to a human.
MAX_RETRIES = 2
for attempt in range(MAX_RETRIES + 1):
resp = agent.step()
if resp.tool_call and not validate(resp.tool_call):
agent.feedback("invalid tool call, retry with correct schema")
else:
break
else:
alert_human(resp)Tradeoff: retries add latency and may loop on a fundamentally impossible request. For latency-sensitive paths, fail closed and surface an error to the user instead of looping. High-risk tools (payment, deletion) should never auto-retry beyond one attempt.
7. Feed production mismatches back into tests
Every rejected call is a free test case. Log the prompt, the raw model output, and the validation error. Weekly, promote a sample into the replay corpus. This closes the loop on detecting hallucinated tool calls AI agents produce under real traffic.
{
"ts": "2025-04-12T10:22:01Z",
"prompt": "ping server 10.0.0.999",
"tool": "restart_server",
"args": {"ip": "10.0.0.999"},
"error": "ip not in allowed subnet"
}Store these in a searchable log, not just a metrics counter. The exact malformed argument is what makes the next test meaningful.
Common pitfalls
Schema too loose. Using type: string for an enum-like field invites fabrication. Use enum or pattern. A field like region should be an explicit list, not free text.
Trusting the model’s self-correction. Some models will return a natural-language excuse (“I meant send_email”). Do not parse excuses; rely on the validator. If the retry still fails, escalate.
Non-determinism masking bugs. Without pinned models or recorded responses, a flaky test passes today and fails after a provider update. Record cassettes and pin versions in CI.
Cost of full replay. Running the live model on a 10k transcript corpus gets expensive. Use recorded cassettes for CI and live runs nightly or weekly.
Ignoring semantic validity. A call can match schema but reference a deleted resource. Add a pre-flight existence check in the mock for high-risk tools, and log when the check fails.
Over-validating. Rejecting a call that a newer model produces correctly because your pattern is too strict creates friction. Review schema changes when model behavior improves.
Where to start
Pick one agent, extract its tool schemas, and add the boundary validator from section 2 this week. Then capture a hundred real prompts and run them through the mock. You will find hallucinations within the first hour. That is the point: detecting hallucinated tool calls AI agents make is a solvable engineering problem, not a research topic. The teams that ship reliable agents are the ones who treat tool calls as hostile input and test accordingly.