How to spot hallucinated code in LLM pull requests

Hallucinated code in an LLM pull request ships silently when reviewers trust the diff. A hallucinated code llm pull request typically introduces functions, APIs, or imports that never existed in the codebase or the referenced libraries, and catching it requires more than a glance at the unified diff.

The failure mode is consistent: the model fills a gap with a plausible-looking symbol that matches the naming conventions of your stack. Below is an end-to-end review workflow that finds those gaps before they reach production.

Step 1: Build an API and dependency inventory before review

You cannot flag a phantom import without a ground-truth list of what actually exists. Generate a frozen dependency set and a symbol index for your environment as part of your baseline CI artifact.

pip freeze > requirements-locked.txt
python -c "import pkg_resources; print([d.project_name for d in pkg_resources.working_set])" > installed_packages.txt

For internal modules, extract the public surface with a quick AST walk. This script lists every top-level function and class in your source tree so you can diff it later:

import ast, pathlib, json

def public_symbols(root):
    syms = {}
    for path in pathlib.Path(root).rglob("*.py"):
        tree = ast.parse(path.read_text())
        for node in tree.body:
            if isinstance(node, (ast.FunctionDef, ast.ClassDef)):
                syms.setdefault(str(path), []).append(node.name)
    return syms

if __name__ == "__main__":
    print(json.dumps(public_symbols("src/"), indent=2))

Commit the output as api_inventory.json. When a PR touches a file, diff the new symbols against the old inventory. Any new import from an external package not present in installed_packages.txt is an immediate candidate for a hallucinated code llm pull request. In a monorepo with multiple languages, repeat the extraction per ecosystem: go doc -all, npm ls, or cargo tree give you the same ground truth.

Step 2: Run static analysis to catch undefined symbols and fake imports

Static checkers catch the cheapest hallucinations: undefined names and unused imports that point to nothing. ruff is fast and has explicit rules for this.

pip install ruff
ruff check --select F821,F401,F811 path/to/changed_files/

F821 flags undefined names. If the PR adds from fastapi import WebSocketAuth and that symbol does not exist in your installed FastAPI version, ruff fails the check. F401 catches imports that are never used—common when the model invents a library to solve a problem it could not otherwise close.

For TypeScript, tsc --noEmit performs the same service. A hallucinated import { useMagic } from 'react' will error because the export is missing. For Go, go build ./... rejects undefined identifiers at compile time.

Do not stop at lint. A hallucinated code llm pull request often imports a real package but calls a method that does not exist. Static analysis will not catch stripe.Customer.create_v2() if stripe is installed. You need runtime attribute checks (Step 4). Treat a clean lint run as necessary, not sufficient.

Step 3: Execute the existing test suite and add characterization tests

A green test suite is necessary but not sufficient. The new code may be entirely untested. Write a characterization test that simply imports and calls the new entrypoint with mocked dependencies. The goal is to force the code to execute the paths the model wrote.

def test_pr_new_helper():
    from myproject.llm_generated import summarize_text
    # If the function internally calls a nonexistent SDK method,
    # it raises AttributeError at runtime, not at import.
    result = summarize_text("sample input")
    assert isinstance(result, str)

Run the suite under pytest -q. If the test fails with AttributeError: module 'x' has no attribute 'y', you have found the hallucination. Add this test to the repo so the gap stays covered.

A hallucinated code llm pull request also frequently invents environment variables or configuration keys. Assert they are not silently assumed:

import os
def test_config_loading():
    # The model referenced os.getenv("FEATURE_FLAG_UNREAL") in the diff
    assert os.getenv("FEATURE_FLAG_UNREAL") is None

Delete the code that reads it. The test now documents that the key was never real.

Step 4: Cross-check external API calls against live objects

For any third-party SDK call, verify the symbol exists in the installed version. A two-line Python check beats a documentation dive and runs in milliseconds.

import stripe
assert hasattr(stripe.Customer, "create"), "Real method missing"
assert not hasattr(stripe.Customer, "create_v2"), "Hallucinated method detected"

If the PR targets an HTTP API, compare the path against the provider’s OpenAPI spec locally:

curl -s https://raw.githubusercontent.com/stripe/openapi/master/openapi/spec3.json -o spec.json
jq '.paths | keys' spec.json | grep "/v2/customers" || echo "Path not in spec"

Replace the URL with your provider’s official spec. The point is mechanical verification: a hallucinated code llm pull request will reference /v2/customers when only /v1/customers exists. Pin your dependency versions in requirements-locked.txt so the hasattr check validates against the exact build that will ship.

Step 5: Use multi-model review to surface inconsistencies

A single model will not reliably flag its own hallucination. Run the review prompt through two different models and diff the complaints. If Model A says “this imports a nonexistent pandas function” and Model B stays silent, inspect that line manually.

Routing the review prompt through n4n.ai lets you honor client routing directives to pin different models per check, and its automatic fallback prevents a single provider outage from stalling your CI. The OpenAI-compatible endpoint works with existing tooling:

from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="KEY")

def review_with(model, diff):
    return client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": f"Find hallucinated symbols in:\n{diff}"}]
    )

for m in ["openai/gpt-4o-mini", "anthropic/claude-3.5-sonnet"]:
    print(m, review_with(m, pr_diff).choices[0].message.content)

If the models disagree on whether pd.read_parquet_v2 exists, a quick hasattr(pd, "read_parquet_v2") settles it. This step turns silent hallucinations into explicit review comments and creates a paper trail for the reviewer.

Step 6: Enforce CI gates and human sign-off

Automate the previous steps in a workflow. A minimal GitHub Actions job that blocks merge on failure:

name: pr-guard
on: [pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.12"}
      - run: pip install ruff pytest stripe
      - run: ruff check --select F821,F401 .
      - run: pytest -q
      - run: python verify_apis.py

Require a reviewer to confirm they ran the multi-model check or reviewed its output. Add a PR template checkbox: “I verified all new external API calls against installed packages.”

A hallucinated code llm pull request loses its stealth when the merge button is gated on these signals. The CI job is the mechanical backstop; the checkbox is the human acknowledgment that the symbols were checked against reality.

Verify success

You have cleaned the PR when:

• ruff reports zero F821/F401 issues on changed files.
• The test suite passes, including new characterization tests that execute the new code paths.
• hasattr checks for every external SDK method referenced in the diff return true.
• The multi-model review produces no unresolved “symbol not found” warnings.
• CI is green and the reviewer checkbox is ticked.

That state does not prove the code is correct, but it proves the symbols exist. In practice, this workflow eliminates the entire class of bugs where the model invented an API that never shipped. Reviewing generated code is not about trusting the model less; it is about trusting the diff only after it survives mechanical checks.