Citation hallucination: when AI invents fake sources

AI citation hallucination fake sources occurs when a language model generates references that appear authoritative — complete with realistic author names, publication venues, DOIs, and page ranges — but do not exist in any verifiable corpus. The model is not retrieving from a database; it is predicting token sequences that statistically resemble citation formats. This distinction matters because the output passes superficial validity checks while being fundamentally ungrounded.

How citation hallucination works

Language models learn the syntax of citations during pretraining. They see millions of patterns like “Smith et al. (2023) demonstrated that…” or “arXiv:2301.12345” and internalize the structural regularities: author-name conventions, year ranges, venue abbreviations, DOI prefixes (10.xxxx), and formatting styles (APA, IEEE, Chicago). At inference time, when prompted for citations, the model samples from this learned distribution.

The failure mode is not random noise. The model produces coherent fabrications because the citation manifold in its representation space is dense and smooth. A nonexistent paper titled “Transformer Architectures for Long-Context Reasoning” with authors “Chen, Liu, and Patel” published in “NeurIPS 2023” sits near real papers in embedding space. The model has no mechanism to distinguish “high-probability citation-like string” from “verified bibliographic entry.”

Two architectural factors amplify this:

1. No retrieval grounding by default. Standard completion endpoints have no access to external knowledge bases unless explicitly augmented with RAG.
2. Training objective mismatch. Next-token prediction rewards local coherence, not global factual consistency. The loss function does not penalize inventing a DOI that checksums correctly but resolves to nothing.

# What the model effectively does (simplified)
def generate_citation(context: str) -> str:
    # Learned pattern: [Author et al., Year] + [Title] + [Venue] + [DOI]
    # No lookup, no verification, no knowledge of what exists
    return sample_from_citation_distribution(context)

Why it matters for production systems

Engineers integrating LLMs into research assistants, legal review tools, medical summarizers, or compliance pipelines face concrete risks:

Downstream contamination. A hallucinated citation fed into a retrieval system as a query term poisons the results. The retriever searches for a paper that does not exist, wasting latency budget and returning irrelevant neighbors.

Cascading trust. Systems that chain LLM calls — e.g., “summarize this paper” → “extract key claims” → “find supporting evidence” — amplify errors. If step one invents a source, step three confidently retrieves other hallucinated sources that “support” the original fabrication.

Regulatory and liability exposure. In regulated domains, presenting fabricated evidence as real can violate rules on accurate record-keeping, professional conduct, or consumer protection. A legal brief citing a nonexistent precedent is a career-ending error.

Evaluation blind spots. Standard benchmarks (MMLU, GPQA) test knowledge retrieval, not citation fidelity. A model can score highly while inventing every reference in a literature review.

Concrete example

Prompt: “Provide three peer-reviewed citations for the claim that chain-of-thought prompting improves mathematical reasoning in LLMs under 7B parameters.”

Model response (fabricated):

1. Wei, J., Wang, X., Schuurmans, D., et al. (2022). “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” Advances in Neural Information Processing Systems, 35, 24824–24837. https://doi.org/10.48550/arXiv.2201.11903

2. Kojima, T., Gu, S.S., Reid, M., et al. (2023). “Large Language Models are Zero-Shot Reasoners.” International Conference on Learning Representations. https://doi.org/10.48550/arXiv.2205.11916

3. Zhang, Y., Li, H., & Chen, M. (2023). “Chain-of-Thought Scaling Laws for Sub-7B Models.” Proceedings of the 40th International Conference on Machine Learning, 202, 41234–41249. https://doi.org/10.48550/arXiv.2306.14287

Citations 1 and 2 are real (the DOI checksums validate and resolve). Citation 3 is a hallucination: the authors are plausible, the venue (ICML 2023) and volume (202) match the real proceedings structure, the page range follows ICML formatting, and the DOI prefix (10.48550) is the correct arXiv prefix. But arXiv.2306.14287 resolves to a different paper entirely — or nothing at all.

An engineer verifying only the first two citations might assume the third is equally valid. This is the danger: hallucinated citations are adversarially plausible.

Detection strategies

1. DOI and identifier verification

Every DOI, arXiv ID, PMID, and ISBN has a checksum or resolution endpoint. Automate verification:

import httpx
import re

DOI_PATTERN = re.compile(r"10\.\d{4,9}/[-._;()/:A-Z0-9]+", re.IGNORECASE)
ARXIV_PATTERN = re.compile(r"arXiv:(\d{4}\.\d{4,5}(v\d+)?)", re.IGNORECASE)

async def verify_doi(doi: str) -> bool:
    """Return True if DOI resolves to a metadata record."""
    url = f"https://doi.org/{doi}"
    async with httpx.AsyncClient(follow_redirects=True, timeout=10.0) as client:
        resp = await client.head(url, headers={"Accept": "application/vnd.citationstyles.csl+json"})
        return resp.status_code == 200

async def verify_arxiv(arxiv_id: str) -> bool:
    """Query arXiv API for existence."""
    url = f"http://export.arxiv.org/api/query?id_list={arxiv_id}"
    async with httpx.AsyncClient(timeout=10.0) as client:
        resp = await client.get(url)
        return "<entry>" in resp.text

async def verify_citation_identifiers(text: str) -> dict:
    """Extract and verify all DOIs and arXiv IDs in a citation block."""
    dois = DOI_PATTERN.findall(text)
    arxiv_ids = [m[0] for m in ARXIV_PATTERN.findall(text)]
    
    results = {"dois": {}, "arxiv": {}}
    for doi in dois:
        results["dois"][doi] = await verify_doi(doi)
    for aid in arxiv_ids:
        results["arxiv"][aid] = await verify_arxiv(aid)
    return results

This catches the obvious fabrications. It does not catch real DOIs attached to wrong papers (citation 3 above would fail because the DOI resolves to a different title).

2. Title-author-venue cross-check

Query scholarly APIs (Crossref, Semantic Scholar, OpenAlex) with the extracted bibliographic tuple:

async def crossref_lookup(title: str, author: str, year: int) -> dict | None:
    """Search Crossref for matching record."""
    params = {
        "query.title": title,
        "query.author": author,
        "query.year": year,
        "rows": 5
    }
    async with httpx.AsyncClient(timeout=10.0) as client:
        resp = await client.get("https://api.crossref.org/works", params=params)
        data = resp.json()
        items = data.get("message", {}).get("items", [])
        return items[0] if items else None

Match on fuzzy title similarity (Levenshtein or embedding cosine) and author overlap. Flag citations where the best match falls below threshold.

3. Retrieval-augmented verification

If you already run a RAG pipeline over a scholarly corpus (Semantic Scholar, PubMed, your internal PDF store), use it as a verifier:

async def verify_via_rag(citation_text: str, rag_query_fn) -> bool:
    """Check if citation content exists in grounded corpus."""
    # Extract key claim + citation metadata as query
    query = extract_verification_query(citation_text)
    results = await rag_query_fn(query, top_k=3)
    # Heuristic: at least one result matches title + first author + year
    return any(
        fuzzy_match(r.title, citation_text) and 
        fuzzy_match(r.authors[0], citation_text) and
        r.year == extract_year(citation_text)
        for r in results
    )

This is the strongest signal but requires maintaining a current scholarly index.

4. Self-consistency probing

Prompt the model multiple times with the same request at temperature > 0. Real citations are stable; hallucinations vary.

async def consistency_check(prompt: str, n: int = 5, temp: float = 0.7) -> float:
    """Return fraction of runs that produce identical citation sets."""
    citations_sets = []
    for _ in range(n):
        resp = await llm_complete(prompt, temperature=temp)
        citations = extract_citations(resp)
        citations_sets.append(frozenset(citations))
    
    # Most common set frequency
    from collections import Counter
    counts = Counter(citations_sets)
    return counts.most_common(1)[0][1] / n

A consistency score below 0.6 suggests fabrication. This is cheap but noisy — some real citations have multiple valid formatting variants.

Mitigation in production

Grounded generation (RAG-first)

The only reliable prevention is retrieving before generating. Architecture:

User query → Retriever (scholarly corpus) → Top-k passages → Generator (cited-only mode) → Output with inline docids

The generator receives a system instruction: “You may only cite sources from the provided context. Use the format [docid]. If the context does not support a claim, say so.”

SYSTEM_PROMPT = """You are a research assistant. You have access to the following documents:
{context}

Rules:
1. Every factual claim must be followed by a citation in square brackets referencing a docid, e.g., [doc_3].
2. If the documents do not contain information to answer the query, respond: "Insufficient evidence in provided sources."
3. Do not hallucinate citations. Do not reference papers not in the context."""

This shifts the burden to retrieval quality. Invest in dense retrieval (BGE, E5, or domain-specific embeddings), hybrid search (BM25 + dense), and reranking (cross-encoder or LLM-based).

Citation-constrained decoding

If you control the model serving stack, constrain the tokenizer to only emit citation tokens that correspond to retrieved docids. This requires a custom logits processor:

class CitationConstraintLogitsProcessor:
    def __init__(self, allowed_citation_tokens: set[int]):
        self.allowed = allowed_citation_tokens
    
    def __call__(self, input_ids, scores):
        # Detect if we're inside a citation bracket [...]
        if self._in_citation_bracket(input_ids):
            # Mask all tokens except allowed citation tokens + closing bracket
            mask = torch.full_like(scores, -float("inf"))
            for tok in self.allowed:
                mask[:, tok] = 0
            mask[:, CLOSE_BRACKET_TOKEN] = 0
            scores = scores + mask
        return scores

This is aggressive but eliminates the syntax-level hallucination entirely. The model can still misattribute claims to wrong docids, but it cannot invent new citation strings.

Post-generation audit pipeline

Treat citation verification as a required CI/CD step for any LLM feature that emits references:

# .github/workflows/citation-audit.yml
name: Citation Audit
on: [pull_request]
jobs:
  verify-citations:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run citation verification
        run: |
          python -m citation_audit \
            --input generated_outputs.jsonl \
            --threshold 0.95 \
            --apis crossref,semantic_scholar,arxiv \
            --fail-on-hallucination

The audit step should be non-negotiable for regulated domains.

Common misconceptions

“Larger models hallucinate less”

Scale improves knowledge retrieval when the knowledge is in the training data. It does not give the model a truth oracle. A 70B model produces more convincing fabrications — better formatting, more plausible author combinations, fewer syntactic tells. The hallucination rate for out-of-distribution citations (recent papers, niche venues) may actually increase because the model’s citation manifold is denser.

“Chain-of-thought prevents citation hallucination”

CoT improves reasoning given correct premises. It does not verify external facts. A model can reason perfectly from a hallucinated premise:

Step 1: I need a paper on CoT scaling for small models. Step 2: Zhang et al. 2023 at ICML sounds right — that venue publishes scaling laws. Step 3: The title “Chain-of-Thought Scaling Laws for Sub-7B Models” fits the pattern. Step 4: DOI 10.48550/arXiv.2306.14287 follows the arXiv prefix convention. Conclusion: Here is the citation.

The reasoning trace looks impeccable. The output is still fiction.

“Fine-tuning on citation tasks fixes this”

Supervised fine-tuning on (prompt, cited-response) pairs teaches the model format compliance, not verification. The training data itself contains hallucinated citations if the annotators didn’t verify every reference (they rarely do). RLHF with a verification reward model helps but requires a reward model that can actually check citations — which brings you back to the retrieval/verification infrastructure.

“If the DOI resolves, the citation is real”

A DOI resolving only proves the identifier exists. It does not prove the bibliographic metadata (title, authors, venue, year) matches what the model claimed. Citation 3 in the example above uses a real DOI prefix and valid checksum structure but points to a different paper. Always verify the full tuple, not just the identifier.

“This only happens with obscure topics”

High-profile topics are more vulnerable because the model has seen more citation patterns for them and the manifold is denser. “Attention Is All You Need” has thousands of citing papers in the training data. The model can interpolate a plausible-sounding 2024 follow-up by “Vaswani et al.” that never happened. Obscure topics sometimes have fewer hallucinations simply because the model falls back to “I don’t know” rather than sampling from a rich citation submanifold.

Practical checklist for engineers

Closing note

Citation hallucination is not a “bug” in the sense of a logic error — it is the model faithfully executing its training objective: produce text that looks like citations. The fix is not better prompting. It is architectural: separate retrieval (what exists) from generation (how to synthesize), and verify the boundary between them. Every production system that emits citations without a verification layer is shipping a known defect.