Catching hallucinated tool arguments before production

LLM agents fail silently when they invent parameters that your API never defined. Detecting hallucinated tool call arguments before production saves you from malformed requests, broken workflows, and angry users.

Step 1: Define strict schemas for every tool

A tool is only as safe as its contract. If you hand the model an open object with no constraints, you are inviting hallucinations. Write a concrete schema for each function the agent can call.

Use JSON Schema or a typed model. I prefer Pydantic for Python services because it gives me both runtime validation and clear errors.

from pydantic import BaseModel, Field
from enum import Enum

class Currency(str, Enum):
    USD = "USD"
    EUR = "EUR"

class ChargeCardArgs(BaseModel):
    amount: float = Field(gt=0)
    currency: Currency
    idempotency_key: str

class RefundArgs(BaseModel):
    charge_id: str
    reason: str | None = None

Export the same shape to the model as a JSON Schema so the prompt and the validator agree:

{
  "name": "charge_card",
  "parameters": {
    "type": "object",
    "properties": {
      "amount": {"type": "number", "exclusiveMinimum": 0},
      "currency": {"type": "string", "enum": ["USD", "EUR"]},
      "idempotency_key": {"type": "string"}
    },
    "required": ["amount", "currency", "idempotency_key"]
  }
}

Keep the schema in one place and generate both the model-facing description and the validator from it. Divergence between the two is how arguments slip through.

Step 2: Validate at the boundary before execution

Never pass raw model output to your business logic. Intercept the tool call, validate, and reject anything that does not fit.

from pydantic import ValidationError
import logging

logger = logging.getLogger("agent.guard")

def dispatch_tool(name: str, args: dict):
    model = TOOL_MODELS.get(name)
    if not model:
        raise ValueError(f"Unknown tool {name}")
    try:
        parsed = model(**args)
    except ValidationError as e:
        logger.warning("hallucinated_args", extra={"tool": name, "args": args, "errors": e.errors()})
        # Return a structured error the agent can recover from
        return {"error": "invalid_arguments", "detail": e.errors()}
    return execute_tool(name, parsed)

This turns silent corruption into a visible signal. The agent can retry with corrected arguments, and you get telemetry on which models hallucinate most.

Capture the rejects

Log every rejected call with the raw args and the validation errors. That log is your training set for later tests.

def log_hallucination(tool, args, err):
    with open("hallucinations.jsonl", "a") as f:
        f.write(json.dumps({"tool": tool, "args": args, "err": err}) + "\n")

Step 3: Record and replay real conversations

Detecting hallucinated tool call arguments at scale requires a corpus of real interactions. Capture the agent’s incoming prompts and its emitted tool calls in a structured transcript.

{"prompt": "refund order #123", "tool": "refund", "args": {"charge_id": "ch_123", "reason": null}}
{"prompt": "charge $50 to USD", "tool": "charge_card", "args": {"amount": 50, "currency": "USD", "idempotency_key": "abc"}}

Store these as JSONL. Then write a replay harness that feeds the prompt to the current model build and compares the emitted args against the recorded ones and the schema.

import json, subprocess

def replay(corpus_path):
    failures = 0
    for line in open(corpus_path):
        rec = json.loads(line)
        out = subprocess.run(["python", "agent_cli.py", rec["prompt"]], capture_output=True, text=True)
        emitted = json.loads(out.stdout)
        if emitted["tool"] != rec["tool"]:
            failures += 1
        else:
            try:
                TOOL_MODELS[emitted["tool"]](**emitted["args"])
            except ValidationError:
                failures += 1
    return failures

If you proxy traffic through a gateway such as n4n.ai, its per-token metering and provider cache-control forwarding give you clean traces to replay without re-incurring cost or hitting rate limits. The replay suite becomes a regression test for model upgrades.

Step 4: Fuzz with adversarial prompts

Real traffic misses edge cases. Generate prompts designed to trick the model into emitting garbage arguments.

ADVERSARIAL = [
    "charge the card but forget the currency",
    "refund with a charge_id that is an integer 12345",
    "call charge_card with amount 'lots' and currency 'dollars'",
    "use refund with no arguments at all",
]

for p in ADVERSARIAL:
    out = subprocess.run(["python", "agent_cli.py", p], capture_output=True, text=True)
    emitted = json.loads(out.stdout)
    try:
        TOOL_MODELS[emitted["tool"]](**emitted["args"])
        print(f"UNCAUGHT: {p} -> {emitted}")
    except ValidationError:
        print(f"blocked: {p}")

Run this against every model version you ship. A model that silently coerces "lots" to 0.0 is worse than one that errors, because it passes your schema but violates intent. Add semantic checks beyond types:

def check_intent(name, args):
    if name == "charge_card" and args["amount"] > 10000:
        raise ValueError("amount exceeds manual review threshold")

Step 5: Wire validation into CI

The replay and fuzz suites belong in your pipeline, not on a notebook. Add a pytest module.

def test_replay_corpus():
    fails = replay("replay.jsonl")
    assert fails == 0, f"{fails} hallucinated calls in replay"

def test_adversarial_blocked():
    for p in ADVERSARIAL:
        out = run_agent(p)
        assert not passes_schema(out), f"prompt slipped: {p}"

Run it on every PR that touches the agent prompt, the tool schemas, or the model pin.

pytest tests/agent_tools.py --cov=agent

Set the build to fail if test_replay_corpus reports any schema violation. That enforces detecting hallucinated tool call arguments as a release gate.

Verify success

Success is concrete: your CI is green on a corpus of at least 200 real transcripts and 50 adversarial prompts, and your production guard logs show zero hallucinated_args warnings over a rolling 7-day window. If you can deploy without manually inspecting tool payloads, the process works.

Step 6: Monitor and close the loop

Validation in pre-prod is necessary but not sufficient. Models drift. Keep the boundary validator in production and alert on validation failures.

if __name__ == "__main__":
    app = make_app()
    app.middleware("http")(guard_middleware)

Feed production rejects back into the replay corpus monthly. The corpus should grow with every new hallucination pattern you discover. Detecting hallucinated tool call arguments is not a one-time audit; it is a continuous contract test between your prompt and your API.

Use the error responses from the guard to fine-tune retries. A good agent will see invalid_arguments and self-correct. Measure correction rate—if it is below 80%, your schema or prompt is unclear, not just the model.

Practical notes

• Keep schemas versioned. A schema change is a breaking API change for the agent.
• Never let the model supply extra keys you ignore; pydantic extra="forbid" catches stray hallucinations.
• Test across providers. The same prompt on two models can yield different argument quality.
• Store the exact provider response when recording; it helps you reproduce flaky hallucinations.

Follow these steps and you will ship agents that call tools correctly or fail loudly—never silently.