A deep research agent hallucinated citations because its architecture asks a language model to write prose and attach references after the fact, not before. The failure is systemic, not a model quirk: when synthesis precedes verification, the generator optimizes for fluency and the citations become decorative. If you are building agentic search and shipping answers to users, you need to treat citation integrity as a data-integrity problem, not a prompt-tuning exercise.
The core mismatch: generation before verification
Standard RAG pipelines retrieve chunks, stuff them in context, and prompt the model to “answer with citations.” This looks correct but inverts the dependency graph. The model produces a token stream left to right. It decides what to say before it has committed to which retrieved span supports it. Even with retrieval context present, the decoder assigns highest probability to sequences that sound authoritative. A citation marker like [1] is just another token, and nothing in the loss function penalizes attaching [1] to a sentence that the referenced chunk does not entail.
In a deep research agent that issues multiple tool calls across turns, the problem compounds. The agent may read ten pages, summarize them in its own memory, and then draft a final report from the summary. By the time the report is written, the original URLs are two hops away from the generated claim. The model is now citing from memory, and memory is not a citation.
How agentic pipelines lose provenance
Retrieval-then-summarize breaks links
Consider a typical research loop:
docs = vector_search(query, top_k=8)
summary = llm.summarize(docs, instruction="Extract key facts")
report = llm.generate(f"Write a report using these facts: {summary}")The summary step is where provenance dies. The summarizer condenses eight documents into free text. Unless you force it to emit structured triples with source IDs, the linkage is lost. The report generator never sees the original chunks; it sees a plausible summary that may already contain drift.
Tool loops that don’t bind sources
A more sophisticated agent uses tools:
for step in range(max_steps):
action = llm.choose_tool(state)
if action == "search":
results = web_search(state.query)
state.add(results) # appends text, not attributed spans
elif action == "answer":
return llm.generate(state.history)If state.add stores raw text without anchoring each fact to a result ID, the final generate call cannot ground its output. The model will still emit [1] because the prompt told it to, but the index maps to a vanished context window.
Why the model itself is biased toward plausible fiction
Transformers are trained on text where citations are sparse and often post-hoc. Academic writing attaches references at the end; web text rarely cites at all. The prior in the weights says “produce confident statements.” When the agent is under retrieval pressure—say a query with few real sources—the model fills gaps with learned patterns of authority. A deep research agent hallucinated citations in our internal eval when the retrieval set was deliberately thin: it invented DOIs that matched the format of real ones but failed resolution.
This is not stupidity. It is the path of least resistance given the objective. Without a hard constraint, the model trades citation accuracy for narrative coherence.
Concrete failure modes
We logged three recurring patterns:
- Format compliance without semantic binding. The model outputs
[1] Source: example.combut the claim preceding it is absent from that page. - Index drift. In multi-turn search, the agent re-indexes results per turn. Citation [3] in the final answer points to turn-2 result 3, but the numbering was reset in turn 3.
- Summarized sources. The agent cites a secondary summary it wrote, not the primary document.
A minimal repro of index drift:
{
"turn_1_results": [{"id": 1, "url": "a.com"}, {"id": 2, "url": "b.com"}],
"turn_2_results": [{"id": 1, "url": "c.com"}],
"final_answer": "Finding X [2]."
}Here [2] is ambiguous: turn-1 id 2 or turn-2 id 2? The agent didn’t namespace.
Engineering defenses that actually work
Constrained decoding and span attribution
Force the model to emit citations as structured objects, not inline markers. Use a schema:
{
"claim": "GPT-4 was released in March 2023.",
"source_id": "doc_7",
"span": "OpenAI announced GPT-4 on March 21, 2023"
}Then render the report by joining claims. If a claim lacks a source_id that maps to a retrieved document, drop it or flag it. This shifts the burden from “model remembers to cite” to “pipeline rejects uncited claims.”
Separate verification pass
Run a second model call with the generated claim and the cited source, asking a binary question: “Does the source text entail the claim? Answer yes/no with quote.” This is cheaper than generation and catches most hallucinations. Route the generator and verifier as independent calls; if the verifier model is rate-limited, you want fallback.
When running a citation validator alongside the generator, routing both through a single OpenAI-compatible endpoint that addresses 240+ models with automatic fallback keeps the pipeline resilient if a provider degrades. You avoid a hard dependency on one vendor’s uptime for your trust layer.
Make citations first-class in state
Redefine agent state as a list of attributed facts:
class Fact:
text: str
source_url: str
retrieved_at: int
state.facts.append(Fact(text=snippet, source_url=url, retrieved_at=now))The generator receives only state.facts and is instructed to reference source_url inline. No re-indexing, no drift.
Tradeoffs: latency, cost, refusal rates
Strict attribution increases latency. A verification pass doubles model calls. Structured extraction may require a more capable (and expensive) model to format output reliably. You will also see refusal rates climb: agents that cannot find a source for a salient point will either stay silent or explicitly say “no source found.” That is correct behavior, but product teams often interpret it as lower recall.
The alternative—fluent reports with decorative citations—erodes user trust faster than a missing answer. We measured qualitatively that a single fabricated DOI in a research summary destroys credibility more than three “unknown” hedges.
Takeaway
Stop treating citations as a formatting instruction. A deep research agent hallucinated citations because the system let it generate first and ground later. Bind every claim to a retrieved span at creation time, namespace sources across turns, and verify with a separate model pass. The deep research agent hallucinated citations only when we skipped these constraints; with structured facts and a validator, the same models produced auditable output. Ship the constraint, not the hope.