A debugging checklist for LLM hallucinations

Hallucinations in production LLM systems rarely appear randomly; they surface through specific failure modes in prompting, context handling, or model selection. An actionable llm hallucination debugging checklist treats each phantom fact as a testable signal rather than a black-box mystery. Below are the steps we run when a model serves confident but incorrect output, drawn from shipping gateways and RAG pipelines under real traffic.

1. Reproduce with deterministic settings

Before blaming the model, remove stochasticity. Set temperature to 0 and pass a fixed seed where the provider supports it. If the hallucination disappears at zero temperature, you are dealing with sampling variance, not a structural prompt defect.

resp = client.chat.completions.create(
    model="mistral-large",
    messages=[{"role": "user", "content": prompt}],
    temperature=0,
    seed=42,
)

Run the same input five times. If the bad output is consistent, the bug is in the prompt, context, or model weight path. If it flips, tighten decoding or add a verification step downstream.

2. Separate the model’s prior from supplied context

A common failure is the model ignoring retrieved text and falling back to parametric memory. Build a stripped variant of the prompt where you replace real context with a placeholder like [CONTEXT REDACTED] but keep the instruction identical.

{
  "messages": [
    {"role": "system", "content": "Answer only from provided context."},
    {"role": "user", "content": "Context: [CONTEXT REDACTED]\nQuestion: Who won the 2022 Nobel in physics?"}
  ]
}

If the model still produces a named laureate, the instruction isn’t binding. That tells you the hallucination is a prompt-adherence problem, not a retrieval problem. Strengthen the system prompt or use logit biasing to suppress out-of-context answers.

3. Verify context window and truncation

Long documents get silently clipped. Log the token count of your assembled prompt and compare against the model’s declared max_context_length. Many SDKs will truncate without error.

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
print(len(enc.encode(full_prompt)))  # vs model limit

If your retrieval dumps 20 chunks but the model only sees the first three because of a naive concatenation, the missing evidence looks like hallucination. Use a sliding window or reranker that respects budget.

3.1 Check the response side too

Hallucinations can appear when the model hits max_tokens mid-entity and completes the thought in a later call. Stream and inspect finish reasons.

{"finish_reason": "length", "usage": {"completion_tokens": 512}}

If you see length, the output was cut, not concluded. Raise the limit or compress the system prompt.

4. Audit tool calls and structured outputs

When the model drives functions, a hallucinated argument often looks like a content hallucination. Validate the schema strictly before execution.

const schema = {
  type: "object",
  properties: {
    ticker: { type: "string", pattern: "^[A-Z]{1,5}$" },
    amount: { type: "number", minimum: 0 }
  },
  required: ["ticker", "amount"]
};

If the model returns ticker: "APPLE" instead of AAPL, reject and retry with a corrected example. A strict JSON parser plus a retry loop catches most of these at the edge.

5. Run a parallel oracle model

Cross-check the suspect output against a second model with different training data. A gateway such as n4n.ai that exposes one OpenAI-compatible endpoint for 240+ models lets you swap the backend without changing client code, making cross-model checks trivial.

def check_with_oracle(prompt, answer):
    crit = client.chat.completions.create(
        model="claude-3-5-sonnet",
        messages=[{"role":"user","content":f"Is this correct? {answer}\nBased on: {prompt}"}]
    )
    return "no" not in crit.choices[0].message.content.lower()

If the oracle flags the answer but the primary model is confident, you have a model-specific bias. Route sensitive requests to the more reliable backend or ensemble.

6. Inspect system prompt and cache directives

Providers increasingly honor cache-control hints. A cached system prompt that drifted from your source of truth will produce stale behavior. Verify what actually ships.

curl -H "Authorization: Bearer $KEY" \
  https://api.example.com/v1/messages/meta | jq '.system_cached'

If the cached hash doesn’t match your deployed prompt, invalidate it. Also confirm the model isn’t silently dropping instructions because they appeared after a large blob of ignored text.

7. Build a minimal eval harness

Don’t debug hallucinations anecdotally. Write a set of 20–50 golden questions with known answers and assert on exact match or regex.

assert "2022" in generate("When did the James Webb telescope launch?")

Run this harness on every prompt change. A regression in pass rate localizes the break. Store failures as fixtures for the next round.

7.1 Measure hallucination rate, not just accuracy

Track the fraction of responses that contain a claim not present in context. A simple entailment check via a smaller model works.

{"label": "contradiction", "score": 0.91}

Over time, this metric is more useful than vibes.

8. Capture raw traffic and diff

Log the exact request body and response for failing cases. Diff against a known-good run to spot injected whitespace, unicode normalization, or a mutated few-shot example.

diff good_req.json bad_req.json

We have caught bugs where a template rendered {{user}} as empty, shifting the message roles. The model then hallucinated a user persona. The diff showed it instantly.

9. Apply constrained decoding

For high-stakes fields, don’t let the model free-form. Use grammar-constrained generation (e.g., outlines or guidancce) to restrict output to a known space.

from outlines import models, generate
model = models.openai("gpt-4o-mini")
generator = generate.choice(model, ["paid", "pending", "failed"])
print(generator(prompt))

If the hallucination was an impossible value, constrained decoding eliminates it by construction.

10. Review the retrieval pipeline independently

In RAG, the LLM is often blamed for a retriever that returned the wrong chunk. Evaluate recall@k on the source corpus separate from the model.

recall = len(set(relevant_ids) & set(retrieved_ids)) / len(relevant_ids)

If recall is below 0.8, fix the embedding or index before touching the prompt. A model cannot cite what it was never given.

Summary table

Working through this llm hallucination debugging checklist systematically turns vague complaints into targeted fixes. Most phantom outputs trace back to one of these ten layers, and the ones that don’t usually reveal a new gap worth adding to your own list.