AI hallucination explained: why LLMs make things up

Hallucination in LLMs is the generation of plausible-sounding but factually incorrect or ungrounded output, caused by the model optimizing for token-level probability rather than truth. It is not a bug — it is the default behavior of a next-token predictor trained on a loss function that rewards fluency, not verification. Understanding why LLMs hallucinate explained through the lens of training objectives and inference mechanics is the first step to building systems that tolerate or suppress it.

How the mechanism works

At training time, an LLM learns a conditional probability distribution $P(x_t | x_{<t})$ over a vocabulary. The loss function — typically cross-entropy — penalizes the model for assigning low probability to the next token in the training corpus. The model has no access to a ground-truth oracle during training; it only sees text. If the training data contains “The capital of France is Paris” and “The capital of France is Lyon” (perhaps from a fictional story), the model learns both sequences are valid continuations weighted by their frequency and context.

At inference time, the model samples or greedily decodes from this distribution. There is no internal fact-checking module. The same weights that produce “Paris” for a factual prompt will produce “Lyon” if the prompt subtly shifts toward a fictional framing, or if the model’s attention lands on a spurious pattern in its training data. The model does not “know” facts; it models the statistical regularities of text that asserts facts.

The role of temperature and sampling

Temperature scales the logits before the softmax:

def sample(logits, temperature=1.0):
    scaled = logits / temperature
    probs = torch.softmax(scaled, dim=-1)
    return torch.multinomial(probs, num_samples=1)

Lower temperature sharpens the distribution, making high-probability tokens more dominant. This reduces variance in outputs but does not eliminate hallucination — it just makes the model more confidently wrong when the highest-probability continuation is false. Higher temperature increases diversity and, consequently, the rate of fabrication. Neither setting addresses the root cause: the model has no mechanism to distinguish “this token follows from evidence” from “this token follows from pattern matching.”

Attention does not equal retrieval

A common mental model is that attention “looks up” facts. It does not. Attention computes a weighted sum of value vectors derived from the context window. If the fact is not in the context — either in the prompt or in the model’s parametric memory activated by the prompt — attention cannot retrieve it. Parametric memory is not a database; it is a lossy, compressed representation of training data correlations. When the model “recalls” a fact, it is reconstructing a probable token sequence, not reading a stored record.

Why it matters for production systems

Hallucination is not an academic curiosity. It breaks trust, causes legal liability, and cascades in agentic workflows.

Silent data corruption

In a RAG pipeline, a hallucinated citation looks identical to a real one. Downstream consumers — whether humans or other LLM calls — treat it as ground truth. A single fabricated API parameter in generated code can cause a deployment rollback. A fabricated medical dosage in a summarization task is a safety incident. The cost is not the token spend; it is the debugging time and the erosion of system reliability.

Compounding in multi-step reasoning

Agents that chain LLM calls amplify hallucination probability. If each step has a 5% hallucination rate (optimistic for complex tasks), a 10-step chain has a ~40% chance of at least one fabrication — and that fabrication becomes context for subsequent steps. The error compounds non-linearly because later steps condition on the hallucinated output.

# Simplified compounding model
def chain_hallucination_rate(step_rate, steps):
    return 1 - (1 - step_rate) ** steps

for steps in [1, 3, 5, 10]:
    print(f"{steps} steps: {chain_hallucination_rate(0.05, steps):.1%}")
# 1 steps: 5.0%
# 3 steps: 14.3%
# 5 steps: 22.6%
# 10 steps: 40.1%

Evaluation blind spots

Standard benchmarks (MMLU, GSM8K) measure accuracy on closed-form questions with verifiable answers. They do not measure hallucination rate on open-ended generation, long-form summarization, or code with subtle semantic bugs. A model can score 90% on MMLU and still invent a non-existent Python library in 30% of coding tasks. You must evaluate your task distribution, not the benchmark’s.

Concrete example: the phantom dependency

Consider a prompt asking for a Python snippet to parse a specific date format using a popular library:

Prompt: “Write Python code to parse ‘2024-03-15T14:30:00Z’ using dateutil.”

The model responds:

from dateutil.parser import isoparse

dt = isoparse("2024-03-15T14:30:00Z")
print(dt)

This works. isoparse exists in dateutil.parser. Now change the prompt slightly:

Prompt: “Write Python code to parse ‘2024-03-15T14:30:00Z’ using dateutil’s strict ISO parser.”

The model responds:

from dateutil.parser import strict_iso_parse

dt = strict_iso_parse("2024-03-15T14:30:00Z")
print(dt)

strict_iso_parse does not exist. The model hallucinated a plausible-sounding function name by combining “strict” (from the prompt) with “iso_parse” (a real function in other libraries like ciso8601). The code fails at runtime with ImportError.

Why this happens

1. Pattern completion: The model has seen many import statements of the form from X.parser import Y_parse. The prompt primes “strict” and “ISO”.
2. No execution feedback: During training, the model never executes code. It only predicts tokens that look like valid code.
3. Plausibility over truth: The token sequence strict_iso_parse has high probability given the context because it follows naming conventions the model has learned. The model has no way to verify the symbol exists in the target library version.

This is not a “smart model making a mistake.” It is a next-token predictor doing exactly what it was trained to do: produce statistically probable continuations.

Common misconceptions

“Larger models hallucinate less”

Scale improves capability — the ability to model complex patterns — but does not fundamentally change the objective. A 70B model produces more convincing hallucinations because its parametric memory is richer and its fluency higher. It can fabricate entire API surfaces with consistent naming, docstrings, and usage examples that do not exist. The hallucination rate on factual QA may drop with scale, but the severity and subtlety of fabrications in open-ended tasks often increases.

“RAG eliminates hallucination”

RAG reduces hallucination if the retrieved context contains the answer and the model attends to it and the model does not contradict it with parametric memory. Failure modes:

• Retrieval misses: The relevant chunk is not in the top-k.
• Context ignorance: The model ignores retrieved context because its parametric memory assigns higher probability to a different answer.
• Context contamination: Retrieved chunks contain conflicting or outdated information; the model synthesizes a plausible but wrong fusion.
• Generator drift: In long-form generation, the model drifts from the context after a few paragraphs.

RAG shifts the problem from “model invents facts” to “retrieval + generation pipeline invents facts.” It is a mitigation, not a solution.

“Fine-tuning on facts fixes it”

Supervised fine-tuning (SFT) on question-answer pairs teaches the model to output correct answers for those specific questions. It does not teach a general truth-tracking mechanism. The model memorizes the training pairs. On out-of-distribution prompts, it reverts to parametric memory or pattern completion. Worse, SFT can cause catastrophic forgetting of the base model’s calibration, making the model more confident on wrong answers.

“Chain-of-thought prevents hallucination”

Chain-of-thought (CoT) improves reasoning on tasks where the reasoning steps are valid. It does not prevent the model from hallucinating premises. If step 1 assumes a false fact, steps 2–N will logically derive a false conclusion. CoT makes the hallucination auditable — you can see where the error entered — but it does not reduce the base rate of premise fabrication.

# CoT example with hallucinated premise
prompt = """
Question: What is the population of Springfield, the capital of Illinois?
Reason step by step.
"""

# Model response:
# Step 1: Springfield is the capital of Illinois. (TRUE)
# Step 2: Springfield has a population of 500,000. (HALLUCINATED - actual ~115k)
# Step 3: Therefore, the population is 500,000.

The reasoning structure is valid; the premise in Step 2 is fabricated. CoT exposes the error but does not prevent it.

“Hallucination is a solved problem with citations”

Forcing the model to cite sources (e.g., [doc_3]) creates a citation generation task. The model learns to emit citation markers that look correct. It can cite a real document for a claim the document does not support, or cite a hallucinated document ID. Verification requires a separate check — either a second model call or deterministic string matching — which adds latency and cost. The citation is not a guarantee; it is a claim that must be verified.

What actually helps (partial list)

The engineering mindset

Stop asking “how do I stop the model from hallucinating?” Start asking:

1. What is the cost of a hallucination in this specific workflow? (User annoyance vs. data corruption vs. safety incident)
2. Where in the pipeline can I verify, constrain, or catch fabrications deterministically?
3. Can I restructure the task so the model only does what it’s reliable at — pattern matching, formatting, style transfer — and delegate facts to a verifiable source?
4. What is my evaluation harness measuring, and does it correlate with production failure modes?

The model will always hallucinate. Your system’s job is to make hallucination detectable, containable, or irrelevant to the outcome. That is a design problem, not a model problem.