RAG pipeline latency: single-query vs multi-hop retrieval

Multi-hop retrieval latency RAG architectures add iterative search rounds to answer questions that span multiple documents, but that iteration directly taxes your tail latency budget. Single-query retrieval sends one embedded question to the vector store and feeds the top-k chunks to the model, keeping the path short and predictable. The choice between these two patterns determines whether your system meets interactive latency targets or collapses under sequential round-trip overhead.

Capabilities

What each approach actually retrieves

Single-query retrieval treats the user’s prompt as a single information need. You embed it once, hit the vector index, and return the nearest neighbors. This works when the answer sits in one or two contiguous passages.

Multi-hop retrieval decomposes the task. The system generates an intermediate query, retrieves, then generates another query conditioned on what it just pulled. This lets it bridge disjoint facts: “Who founded the lab that published the 2023 fusion result?” requires finding the paper, then the lab, then the founder.

The capability gap is real, but multi-hop does not guarantee correctness. Each hop inherits the prior hop’s mistakes. On multi-hop benchmarks such as HotpotQA, iterative retrieval closes a meaningful recall gap that single-shot embedding search cannot, but only when the rewriter model is prompt-tuned for the domain.

# single-query: one round trip to the index
def single_rag(query, embedder, db, llm, k=8):
    vec = embedder.encode(query)
    hits = db.search(vec, top_k=k)
    ctx = "\n".join(h.text for h in hits)
    return llm.complete(f"Context:\n{ctx}\n\nQ: {query}")

# multi-hop: model-driven exploration
def multi_hop_rag(query, embedder, db, llm, max_hops=3):
    collected = []
    sub_q = query
    for _ in range(max_hops):
        vec = embedder.encode(sub_q)
        hits = db.search(vec, top_k=4)
        collected.extend(hits)
        next_q = llm.complete(
            f"Original: {query}\nContext: {collected}\n"
            "Write next sub-query or 'STOP'."
        )
        if next_q.strip() == "STOP":
            break
        sub_q = next_q
    ctx = "\n".join(h.text for h in collected)
    return llm.complete(f"Context:\n{ctx}\n\nQ: {query}")

Price/cost model

Single-query cost is easy to bound: one embedding call, one completion. The context size is fixed at k chunks.

Multi-hop multiplies both. You pay for an embedding per hop, a completion to rewrite the query per hop, and a final completion over an accumulating context. If each hop adds 4 chunks of 512 tokens, three hops inject ~6k retrieved tokens before generation. That context inflation hits every token-metered provider. Embedding models are cheap—typically fractions of a cent per 1k tokens—but the generation tokens dominate. Multi-hop’s hidden tax is the rewriter completion, which often uses the same expensive model as the final answer.

If you route through a unified gateway such as n4n.ai, per-token usage metering exposes the exact surcharge per hop, letting you cap loops before they burn budget. Without that visibility, teams routinely ship multi-hop loops that cost 5x a single query for a 10% accuracy gain.

Latency/throughput

Retrieval itself is cheap at scale. ANN indexes return top-k in single-digit to low-double-digit milliseconds on million-vector sets. Embedding a sentence is sub-50ms on CPU or GPU batch. The dominant cost is LLM generation.

Single-query latency is one embedding + one search + one generation. On a 70B-class model at 30 tok/s, a 300-token answer takes ~10s, but the retrieval portion is noise.

Multi-hop serializes generation and retrieval. Hop 1: embed (30ms) + search (10ms) + rewrite gen (500ms). Hop 2 repeats. Final gen (10s). You have added 1–2s of sequential LLM calls before the answer even starts. You can parallelize the embedding and search of hop 2 while the rewriter thinks, but the dependency on the rewriter’s output for the next query prevents full overlap. Pipeline depth, not throughput, is the killer. Under load, queueing worsens tail latency. Throughput per GPU drops because each request holds a session across multiple inferences.

# rough p95 observation from a 1M-vector index + local 7B model
single-query:  embed 40ms | search 12ms | gen 4200ms => ~4.3s
multi-hop(3):  +2*(embed 40 + search 12 + rewrite 600) => +1.3s, gen same => ~5.6s
# numbers are environment-specific; treat as order-of-magnitude

Ergonomics

Single-query is a function call. You can cache the embedding, reuse the prompt, and unit-test the output.

Multi-hop forces you to manage state: the growing context window, a stop condition, parse failures from the rewriter, and timeout per hop. The rewriter model will occasionally emit malformed queries or loop. You need guardrails. Observability tools that trace a single request suddenly need to represent a tree of sub-queries. A gateway that honors client routing directives and forwards provider cache-control hints—such as n4n.ai—lets you mark the static system prompt and early context as cacheable, reducing recompute across hops.

// minimal stop logic in TS
async function multiHop(query: string, hops: number) {
  let sub = query;
  const ctx: string[] = [];
  for (let i = 0; i < hops; i++) {
    const vec = await embed(sub);
    const hits = await search(vec, 4);
    ctx.push(...hits.map(h => h.text));
    const next = await llm(`Context: ${ctx}\nNext sub-query or STOP`);
    if (next.trim() === "STOP") break;
    sub = next;
  }
  return llm(`Context: ${ctx}\nQ: ${query}`);
}

Debugging a multi-hop failure means inspecting each intermediate query. That is a different operational posture than a straight pipeline.

Ecosystem

Every RAG framework supports single-query out of the box. LangChain’s VectorStoreRetriever, LlamaIndex’s VectorIndexRetriever, and raw SDK calls all assume one search.

Multi-hop exists as opt-in: LangChain’s MultiQueryRetriever fans out parallel queries (not strictly sequential hops), while AgentExecutor with a search tool approximates true multi-hop. LlamaIndex offers CitationQueryEngine and iterative modules. You will write custom loop code in either case; the primitives are there but the orchestration is on you. Neither pattern benefits from model-endpoint diversity unless you explicitly swap the rewriter model to a smaller one; that’s a tuning lever, not a framework feature.

Limits

Single-query fails on questions requiring joining facts across corpora. It also suffers when the user query is lexically distant from the answer text (poor embedding recall).

Multi-hop suffers from error compounding: a bad first hop biases all later ones. It has no inherent fact-checking. Latency and cost scale with hop count, and nothing prevents the model from hopping forever without a hard cap. For compliance-sensitive logs, the expanding context may exceed redaction boundaries. Multi-hop also complicates compliance: the union of retrieved chunks may cross document access boundaries that a single query would respect.

Head-to-head summary

Dimension	Single-query	Multi-hop retrieval
Capabilities	One-shot top-k; good for local context	Iterative decomposition; bridges disjoint facts
Price/cost model	1 embed + 1 gen; fixed context tokens	N embeds + N rewrites + 1 gen; growing context
Latency/throughput	One gen bound; retrieval negligible	Sequential gens per hop; p95 grows linearly
Ergonomics	Trivial to test and cache	Stateful loop, stop logic, parse handling
Ecosystem	Native in all frameworks	Opt-in agents or custom loops
Limits	Poor recall for multi-entity queries	Error propagation, unbounded cost/latency

Which to choose

Use single-query when

Your documents are chunked so answers live in 1–2 neighbors.
Latency SLO is strict (sub-5s interactive).
Cost per request must stay flat at high traffic.
You can improve recall with hybrid search (BM25 + vector) instead of more hops.

Use multi-hop retrieval latency RAG when

Questions are explicitly compositional (“X of Y’s Z” across sources).
A human is waiting but tolerates 5–10s for a researched answer.
You have eval data showing single-query recall falls short on target queries.
You implement a hard max_hops and monitor per-hop token spend.

Hybrid pattern

In production, start single-query. Add a classifier that routes only ambiguous, multi-entity queries to a capped multi-hop path. This contains the latency and cost blowup while preserving coverage. The multi-hop retrieval latency RAG tax is acceptable only on the fraction of traffic that needs it.