Most RAG pipelines break on questions that require connecting two or more disjoint facts. A multi-hop retrieval agentic RAG approach treats retrieval as a stateful loop where the agent inspects accumulated context and decides the next fetch, rather than embedding the original query once and hoping the top-k chunks suffice.
Why single-shot retrieval fails on branched queries
A question like “Which vendor supplied the capacitor used in the drone that crashed in 2023?” contains three implicit subqueries: identify the crash report, extract the capacitor part number, then resolve the supplier. A flat vector search over the original string returns a mix of crash reports, capacitor datasheets, and vendor catalogs, ranked by surface similarity. The model gets noise and misses the chain.
Multi-hop retrieval agentic RAG fixes this by letting the LLM drive the lookup path. Each hop narrows the search space using facts from the previous hop. This is not a bigger embedding model; it is control flow.
Step 1: Model the agent state explicitly
Before writing any LLM calls, define what the agent carries between hops. A typed state prevents the loop from silently losing context or repeating searches.
from dataclasses import dataclass, field
from typing import List, Dict
@dataclass
class AgentState:
query: str
facts: List[Dict] = field(default_factory=list)
subquestions: List[str] = field(default_factory=list)
hops: int = 0
max_hops: int = 5
sources: List[str] = field(default_factory=list)The subquestions list is the key lever. After the first hop, the model should populate it with what it still needs. If it stays empty and no answer is derivable, you terminate with “insufficient data” instead of guessing.
A common mistake is storing only raw text in facts. Store provenance too: {"text": "...", "source": "doc_482", "hop": 1}. You will need it for citations and debugging.
Step 2: Expose retrieval as strict tools
The agent must call real functions, not imagine a search happened. Define OpenAI-compatible tool schemas and implement the backing retriever separately.
{
"type": "function",
"function": {
"name": "search_kb",
"description": "Semantic search over the engineering doc store. Returns chunk ids and text.",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string"},
"top_k": {"type": "integer", "default": 3}
},
"required": ["query"]
}
}
}Add a second tool for fetching a full document by ID, and optionally a query_sql tool if structured metadata lives in Postgres. Keep the descriptions opinionated: “Use only after you have a part number” beats vague phrasing.
The model selects tools based on the schema. If you pass a single “retrieve” catch-all, the planner will hallucinate query strings that span hops and defeat the purpose of the multi-hop retrieval agentic RAG design.
Step 3: Run the planner-executor loop
The loop is a standard ReAct-style cycle with a hard cap. Below is a minimal version using the OpenAI Python client.
from openai import OpenAI
client = OpenAI() # defaults to env var OPENAI_API_KEY
tools = [search_kb_schema, fetch_doc_schema]
state = AgentState(query="Which vendor supplied the capacitor in the 2023 drone crash?")
messages = [{"role": "user", "content": state.query}]
while state.hops < state.max_hops:
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=tools,
tool_choice="auto"
)
msg = resp.choices[0].message
if not msg.tool_calls:
# Model thinks it can answer or needs to stop
break
for call in msg.tool_calls:
result = dispatch(call) # your retriever or DB call
messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
state.facts.append({"text": result, "tool": call.function.name, "hop": state.hops})
state.hops += 1
final = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages + [{"role": "user", "content": "Answer with sources."}]
)The dispatch function must be deterministic and side-effect free for a given input. Log every call with the hop index; when the agent goes in circles, the logs show it immediately.
In a multi-hop retrieval agentic RAG deployment, the model in the loop and the model for final synthesis can differ. Use a cheaper model for planning hops and a stronger one for the final answer if cost matters.
Step 4: Enforce termination and provenance
Unbounded agents burn tokens and latency. Terminate when either:
state.subquestionsis empty and the model returns a non-tool message, orstate.hops == state.max_hops.
Always require the final response to cite sources from state.facts. A simple post-check: parse the answer, verify each cited doc ID exists in state.sources. If not, reject and re-prompt.
def verify_citations(answer: str, state: AgentState) -> bool:
cited = extract_ids(answer)
return all(c in state.sources for c in cited)If verification fails twice, return the raw facts and flag for human review. Do not let the model silently invent a source.
Common pitfalls and tradeoffs
Context bloat
Every hop appends tool results to messages. By hop 4, you may exceed the model window or dilute attention. Mitigate by summarizing facts after each hop: keep a rolling summary field and pass only that plus the last raw result to the next planner call.
Hallucinated subquestions
The planner sometimes generates a subquestion it already answered. Add a guard: before executing a tool, check similarity of the new query against previous facts. If cosine similarity > 0.9, skip and decrement hop.
Latency versus depth
Each hop is at least one round-trip plus retrieval. A 5-hop agent on a 200 ms model call plus 100 ms search is ~1.5 s best case, often 5–10 s with retries. For user-facing chat, cap max_hops at 3 and use parallel tool calls where independent.
Evaluation is not accuracy alone
Log hop paths and build a dataset of multi-hop questions with known chains. Measure “path correctness” (did it fetch the right docs in order) separately from “answer correctness.” A correct answer from a wrong path is a latent failure.
Routing models without losing the loop
When you run this loop across multiple providers, model availability becomes a real failure mode. A rate-limited completion mid-hop throws the whole state away unless you build retry logic. An OpenAI-compatible gateway such as n4n.ai gives you automatic fallback and forwards provider cache-control hints, so a degraded embedding endpoint doesn’t stall the agent. The agent code stays identical; you only change the base_url.
The tradeoff is added network dependency and potential cost opacity. Use per-token metering to attribute spend to each hop, otherwise debugging a sudden bill spike is painful.
Shipping checklist
- State class with facts, subquestions, sources, hop count.
- At least two distinct retrieval tools with precise descriptions.
- Loop with
max_hopsand explicit termination. - Citation verification on final answer.
- Hop-path logging from day one.
- Model routing that survives a single provider outage.
Multi-hop retrieval agentic RAG is not exotic. It is a disciplined state machine wrapped around a model that knows how to ask its own follow-ups. Build the guardrails first; the LLM will handle the branching.