Hallucination in code generation: when AI invents APIs

Hallucination in AI code generation occurs when a language model produces syntactically valid code that references APIs, methods, classes, or parameters that do not exist in the target library or runtime. The model mimics the statistical patterns of real code without grounding in actual documentation or source definitions. This differs from logic errors: the code parses and often type-checks, but fails at runtime with attribute errors, import failures, or silent wrong behavior.

How hallucination in code generation works

Language models predict tokens based on training distribution, not a verified symbol table. When you prompt for “async HTTP client with retry logic in Python,” the model activates patterns from httpx, aiohttp, requests, and urllib3 simultaneously. It blends parameter names (timeout vs read_timeout), method signatures (client.get() vs client.request("GET", ...)), and class hierarchies into a plausible chimera.

The mechanism operates at three levels:

Token-level blending: The model has seen timeout=30.0 in httpx and timeout=aiohttp.ClientTimeout(total=30) in aiohttp. It may emit timeout=httpx.Timeout(30.0) — valid syntax, invalid API.

Structural improvisation: Given a partial pattern like “context manager for database transactions,” the model may invent async with db.transaction(): even when the library only supports async with db.begin(): or requires explicit await db.commit().

Cross-contamination: Training data contains multiple versions of the same library. The model merges v1 and v2 APIs, deprecated and current patterns, producing code that would have worked in 2021 but fails today.

Why it matters for production systems

Hallucinated APIs are the most dangerous class of LLM coding errors because they pass superficial review. The code looks idiomatic. It often passes static analysis if the hallucinated names follow naming conventions. CI/CD pipelines miss them unless integration tests exercise the exact code path.

Consider the failure modes:

• Runtime crashes: AttributeError: module 'redis' has no attribute 'async_cluster' — immediate, visible, debuggable.
• Silent data corruption: A hallucinated cache.set(key, value, ttl=300) that actually maps to cache.set(key, value, expire=300) in the real API. The call succeeds but TTL is ignored. Data expires on default policy.
• Security bypass: Invented sanitize_html(input, strict=True) that doesn’t exist. The real function is sanitize(input, mode="strict"). The call fails silently or falls through to a no-op.
• Cost explosions: Hallucinated batching APIs like embeddings.create_batch() that don’t exist. The model falls back to sequential calls in a loop, multiplying latency and token spend by 100x.

Concrete example: the async Redis client that never existed

A developer asks for “async Redis pipeline with automatic retry in Python.” The model generates:

import redis.asyncio as redis
from redis.retry import Retry
from redis.backoff import ExponentialBackoff

async def get_cached_data(keys: list[str]) -> dict[str, bytes]:
    retry_policy = Retry(ExponentialBackoff(), retries=3)
    client = redis.Redis(
        host="localhost",
        port=6379,
        retry=retry_policy,
        retry_on_timeout=True,
    )
    
    async with client.pipeline(transaction=True) as pipe:
        for key in keys:
            pipe.get(key)
        results = await pipe.execute()
    
    return dict(zip(keys, results))

This code passes ruff, mypy, and a quick mental review. It follows redis-py patterns perfectly. But three hallucinations hide here:

1. redis.retry.Retry and redis.backoff.ExponentialBackoff — these classes exist in redis but not in redis.asyncio. The async client accepts retry as a callable, not a Retry object.
2. retry_on_timeout=True — not a parameter of redis.asyncio.Redis.__init__.
3. client.pipeline(transaction=True) — the async pipeline method signature is pipeline(transaction: bool = True, shard_hint: Any = None), but the context manager protocol wasn’t added until v5.0. Earlier versions require explicit await pipe.execute() without async with.

The corrected version requires checking the installed version and reading source:

import redis.asyncio as redis
from redis.asyncio.retry import Retry
from redis.asyncio.backoff import ExponentialBackoff

async def get_cached_data(keys: list[str]) -> dict[str, bytes]:
    retry_policy = Retry(ExponentialBackoff(), retries=3)
    client = redis.Redis(
        host="localhost",
        port=6379,
        retry=retry_policy,
    )
    
    pipe = client.pipeline(transaction=True)
    for key in keys:
        pipe.get(key)
    results = await pipe.execute()
    
    return dict(zip(keys, results))

The differences are subtle. The hallucinated version would raise TypeError: __init__() got an unexpected keyword argument 'retry_on_timeout' at runtime — but only when the function executes. If this sits behind a feature flag or rare code path, it ships to production.

Common misconceptions

“TypeScript prevents this”

Static typing catches some hallucinations — if the model invents a method on a typed interface, the compiler errors. But:

• Most hallucinations occur in dynamic languages (Python, JavaScript) where the ecosystem dominates LLM training data.
• Even in TypeScript, the model can hallucinate correctly typed but non-existent overloads: fetch(url: string, options: { retry: number }) when the real signature is fetch(url: string, options?: RequestInit).
• Declaration files (.d.ts) for popular libraries are often incomplete or outdated in training data. The model learns stale types.

“RAG with documentation solves it”

Retrieval-augmented generation helps but introduces new failure modes:

• Chunking artifacts: Documentation split into 512-token chunks loses cross-references. The model sees Client.timeout in one chunk and TimeoutConfig in another, merges them incorrectly.
• Version skew: Retrieved docs may not match the pinned dependency version. The model trusts the retrieved text over its parametric knowledge.
• Context pollution: Irrelevant retrieved chunks (e.g., sync API docs when asking for async) increase hallucination probability by diluting the relevant signal.

“Just add ‘only use real APIs’ to the prompt”

Negative constraints (“don’t hallucinate”) are weaker than positive grounding (“here is the exact API surface”). The model has no introspective access to its own knowledge boundaries. It cannot distinguish “I know this API” from “this pattern feels familiar.”

“Unit tests catch hallucinations”

Only if the test exercises the hallucinated path. Most generated tests mock the same hallucinated APIs, creating a self-consistent fantasy. Integration tests against real dependencies are required — but they’re slow, flaky, and often absent for the exact code paths LLMs generate.

Detection strategies that work

1. Symbol verification against installed packages

Parse the generated AST, extract all attribute accesses and imports, verify against the actual installed package’s __all__ and module structure.

import ast
import importlib
import sys

def verify_symbols(code: str, package: str) -> list[str]:
    """Return list of unverified symbols in generated code."""
    tree = ast.parse(code)
    unverified = []
    
    module = importlib.import_module(package)
    valid_attrs = set(dir(module))
    
    for node in ast.walk(tree):
        if isinstance(node, ast.Attribute):
            if isinstance(node.value, ast.Name):
                if node.value.id == package and node.attr not in valid_attrs:
                    unverified.append(f"{package}.{node.attr}")
        elif isinstance(node, ast.ImportFrom):
            if node.module and node.module.startswith(package):
                for alias in node.names:
                    try:
                        importlib.import_module(f"{node.module}.{alias.name}")
                    except ImportError:
                        unverified.append(f"{node.module}.{alias.name}")
    
    return unverified

This catches redis.asyncio.Retry immediately. It misses version-specific symbols and dynamic __getattr__ exports.

2. Dry-run execution in isolated environment

Execute generated code against the real dependency in a throwaway container. Capture import errors, attribute errors, and signature mismatches via inspect.

import subprocess
import tempfile
import json

def dry_run(code: str, requirements: list[str]) -> dict:
    with tempfile.TemporaryDirectory() as tmpdir:
        # Write code and requirements
        (Path(tmpdir) / "generated.py").write_text(code)
        (Path(tmpdir) / "requirements.txt").write_text("\n".join(requirements))
        
        # Build and run in isolated container
        dockerfile = f"""
        FROM python:3.11-slim
        WORKDIR /app
        COPY requirements.txt .
        RUN pip install --no-cache-dir -r requirements.txt
        COPY generated.py .
        RUN python -c "
import generated
import json
import inspect
results = {{}}
for name in dir(generated):
    obj = getattr(generated, name)
    if callable(obj):
        try:
            sig = inspect.signature(obj)
            results[name] = str(sig)
        except Exception as e:
            results[name] = f'ERROR: {{e}}'
print(json.dumps(results))
"
        """
        (Path(tmpdir) / "Dockerfile").write_text(dockerfile)
        
        result = subprocess.run(
            ["docker", "build", "-q", tmpdir],
            capture_output=True, text=True, timeout=120
        )
        
        if result.returncode != 0:
            return {"error": "build_failed", "stderr": result.stderr}
        
        image_id = result.stdout.strip()
        run_result = subprocess.run(
            ["docker", "run", "--rm", image_id],
            capture_output=True, text=True, timeout=30
        )
        
        return json.loads(run_result.stdout) if run_result.returncode == 0 else {"error": "runtime_failed"}

Expensive but definitive. Run on every generated module before merge.

3. Differential testing against multiple models

Generate the same task with 3+ models (GPT-4o, Claude 3.5 Sonnet, DeepSeek-Coder). Compare AST structures. Consensus on an API that doesn’t exist in docs is a strong hallucination signal.

def consensus_check(generations: list[str], package: str) -> dict:
    """Find symbols agreed upon by multiple models but missing from package."""
    all_symbols = []
    for gen in generations:
        symbols = extract_symbols(gen, package)
        all_symbols.append(set(symbols))
    
    # Symbols appearing in >=2 generations
    from collections import Counter
    flat = [s for sym_set in all_symbols for s in sym_set]
    counts = Counter(flat)
    consensus = {s for s, c in counts.items() if c >= 2}
    
    # Verify against actual package
    real = get_real_symbols(package)
    hallucinated = consensus - real
    
    return {
        "consensus_symbols": list(consensus),
        "verified": list(consensus & real),
        "likely_hallucinated": list(hallucinated)
    }

4. Provider-aware routing for code tasks

Different models hallucinate differently. GPT-4o leans toward OpenAI SDK patterns. Claude favors Anthropic’s own SDK conventions. DeepSeek-Coder mirrors Hugging Face transformers APIs. Route code generation tasks to the model whose training distribution matches your stack.

MODEL_ROUTING = {
    "openai": "gpt-4o",
    "anthropic": "claude-3-5-sonnet",
    "huggingface": "deepseek-coder-33b",
    "langchain": "gpt-4o",  # heavy OpenAI examples in training
    "redis": "claude-3-5-sonnet",  # better async Python in training
    "sqlalchemy": "deepseek-coder-33b",  # strong ORM patterns
}

def route_code_task(library: str, prompt: str) -> str:
    model = MODEL_ROUTING.get(library.lower(), "gpt-4o")
    return call_model(model, prompt)

This is where a gateway like n4n.ai reduces friction — one endpoint, 240+ models, automatic fallback when a provider degrades, and per-token metering so you can measure which model actually produces fewer hallucinations for your stack.

Prevention workflow for teams

1. Generate with context: Include the target library’s version-pinned pyproject.toml or package.json in the prompt. Paste the relevant __init__.py exports if the library is small.

2. Verify before review: Run symbol verification and dry-run in CI on every PR containing LLM-generated code. Block merge on unverified symbols.

3. Maintain a hallucination registry: Log every caught hallucination with the prompt, model, and invented API. Pattern-match new generations against this registry.

4. Prefer SDKs over raw HTTP: Hallucination rate drops when the model uses official client libraries vs constructing requests manually. The SDK surface is smaller and better represented in training.

5. Constrain output format: Require the model to emit a JSON object with code, imports, and dependencies fields. Parse and validate each field separately before assembling.

{
  "imports": ["redis.asyncio"],
  "dependencies": ["redis>=5.0.0"],
  "code": "async def get_cached_data(keys: list[str]) -> dict[str, bytes]: ..."
}

When to accept risk

Not all hallucinations are equal. A hallucinated convenience method that raises AttributeError on first call is a bug. A hallucinated parameter that silently changes behavior is an incident. Prioritize verification effort by:

• Critical path: Payment processing, auth, data mutation — full dry-run required.
• Read-only helpers: Logging, formatting, display — symbol verification sufficient.
• Exploratory code: Prototypes, one-off scripts — accept risk, verify manually.

The bottom line

Hallucination in AI code generation is not a reasoning failure — it’s a retrieval failure. The model retrieves statistical patterns, not verified symbols. Treat generated code as untrusted input. Verify against ground truth (installed packages, live APIs, type stubs) before it reaches a code reviewer. The reviewer’s job is logic and architecture, not API existence.

The tools exist. The discipline is what’s missing.