How to detect hallucinated function calls in tool use

Hallucinated function calls break agentic workflows silently: the model emits a tool invocation that looks valid but references an undefined function or impossible arguments. To detect hallucinated function calls reliably, you need a validation layer that intercepts the model’s output before execution, plus telemetry to catch drift over time.

Step 1: Define a strict machine-readable tool schema

Start by expressing every tool your agent can call as a JSON Schema. The model should receive these definitions in the standard tools array, but your server must hold the canonical copy. Never let the client or the model dictate what is callable.

{
  "name": "get_weather",
  "description": "Fetch current weather for a location",
  "parameters": {
    "type": "object",
    "properties": {
      "lat": {"type": "number", "minimum": -90, "maximum": 90},
      "lon": {"type": "number", "minimum": -180, "maximum": 180}
    },
    "required": ["lat", "lon"],
    "additionalProperties": false
  }
}

Set additionalProperties: false explicitly. Models will invent parameters like unit when you didn’t define it; this flag turns that into a hard error. If you are writing the agent in Python, back the schema with a Pydantic model so your validation code and your type hints stay in sync.

from pydantic import BaseModel, Field

class GetWeatherArgs(BaseModel):
    lat: float = Field(ge=-90, le=90)
    lon: float = Field(ge=-180, le=180)

Store each tool in a registry keyed by name. Include the JSON Schema and the Pydantic class. This registry is the ground truth you will use in later steps.

Step 2: Intercept the model response before execution

When you call a chat completions API, the response may contain tool_calls. In a production agent, you must inspect every call before touching any side-effecting system. The simplest non-streaming case looks like this:

from openai import OpenAI

client = OpenAI()  # or any OpenAI-compatible endpoint
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    tools=TOOL_DEFINITIONS,
)

Extract the calls:

def extract_tool_calls(resp):
    msg = resp.choices[0].message
    if not msg.tool_calls:
        return []
    return [tc for tc in msg.tool_calls if tc.type == "function"]

If you use streaming, the tool_calls arrive in chunks with index fields. Accumulate arguments per index before validation. A common bug is validating each delta; never do that. Buffer the full call, then check.

If a tool_call names a function absent from your registry, you have detected a hallucinated function call. Log it and skip execution.

Step 3: Validate arguments against the schema

Name matching is not enough. Arguments can be malformed, partially valid, or semantically nonsense. Use jsonschema (or Pydantic) to validate the parsed arguments.

import json
from jsonschema import validate, ValidationError

def validate_call(tc, registry):
    if tc.function.name not in registry:
        return False, f"Unknown function: {tc.function.name}"
    schema = registry[tc.function.name]["parameters"]
    try:
        args = json.loads(tc.function.arguments)
    except json.JSONDecodeError as e:
        return False, f"Invalid JSON: {e}"
    try:
        validate(instance=args, schema=schema)
    except ValidationError as e:
        return False, f"Schema violation: {e.message}"
    return True, ""

Models frequently emit arguments as a string containing trailing commas or markdown fences. Strip those before json.loads. If parsing fails, that is also a hallucination signal—the model could not conform to the contract.

For parallel tool calls, loop over the list. One bad call should not block the others unless your business logic requires atomicity.

Step 4: Log and score every call for hallucination signals

Emit a structured log line for each tool call attempt. Capture the model ID, the function name, the raw argument string, and the validation outcome.

import logging
logging.basicConfig(level=logging.INFO)

def log_attempt(model, tc, ok, reason):
    logging.info({
        "event": "tool_call_attempt",
        "model": model,
        "function": tc.function.name,
        "valid": ok,
        "reason": reason,
        "arguments": tc.function.arguments,
    })

If you route through n4n.ai, its OpenAI-compatible endpoint spans 240+ models with automatic fallback, letting you centralize this validation log regardless of which backend actually served the request. That matters when you A/B test models and need a single telemetry pipe.

Aggregate these logs daily. Compute hallucination rate per model and per tool. A baseline of 0.1% can jump to 5% after a provider silently swaps a model snapshot. Without this step you will only notice when a customer reports a broken action.

Step 5: Run a shadow executor for valid calls

Schema-valid calls can still be contextually hallucinated. Example: get_weather with lat: 40.7, lon: -74.0 is valid, but if the user asked for Tokyo, the agent hallucinated the location mapping. Catch this with a shadow executor that records intent.

def shadow_execute(name, args, user_query):
    # record what the agent thinks it should do
    return {"called": name, "args": args, "query": user_query}

Feed the shadow output into a secondary checker—either a smaller model prompted to verify alignment, or a deterministic rule (e.g., query contains “Tokyo” => lat/lon near 35.6/139.7). If mismatch, flag for human review. This step moves you from syntactic to semantic detection.

Step 6: Build an adversarial regression suite

Detection is only proven if it survives model updates. Create a pytest suite that sends prompts designed to provoke hallucinations: ask for functions you didn’t define, request impossible parameters, or use slang.

import pytest

@pytest.mark.parametrize("prompt", [
    "Use the send_fax tool to message Bob",
    "Get weather at latitude 999",
    "Call the database_drop function",
    "Ping the CRM for last quarter's revenue",
])
def test_no_hallucination(prompt):
    messages = [{"role": "user", "content": prompt}]
    resp = client.chat.completions.create(
        model="gpt-4o-mini", messages=messages, tools=TOOL_DEFINITIONS)
    calls = extract_tool_calls(resp)
    for tc in calls:
        ok, reason = validate_call(tc, REGISTRY)
        assert ok, f"Hallucinated: {reason}"

Run this in CI on every model upgrade and on a nightly cron against production models. When a new prompt slips through, add it to the suite. Over time you build a corpus that reflects real attack surface.

Step 7: Verify success with measurable thresholds

Define success concretely. For production traffic, set:

• Zero unknown function names over any 24-hour window.
• Schema validation failure rate below 0.5% on calls that name known functions.
• Semantic mismatch rate (from shadow mode) below 1%.

Alert when any threshold is crossed.

# Pseudo-SQL alert
SELECT model,
       COUNT(*) FILTER (WHERE valid = false) * 1.0 / COUNT(*) AS halluc_rate
FROM tool_call_logs
WHERE ts > now() - interval '24 hours'
GROUP BY model
HAVING halluc_rate > 0.005;

When the alert fires, pull the offending arguments and the preceding messages from your log store. Replay them in the regression suite to confirm the fix. Success is not “no hallucinations ever”; it is “detected, logged, and routed to a safe fallback every time.”

Common pitfalls

• Case sensitivity: Get_Weather vs get_weather. Normalize names before lookup.
• Ignoring id: tool calls carry an id used in subsequent tool messages. Log it; otherwise you cannot trace a hallucination back to its cause.
• Over-permissive schemas: type: object with no properties accepts anything. Tighten schemas quarterly.
• Single-call assumption: agents emit arrays. Validate all elements.

Following these steps gives you a pipeline that can detect hallucinated function calls at the edge, measure them, and starve them of production side effects.