Multi-document RAG with LlamaIndex RouterQueryEngine

You need to query across multiple document collections — maybe a codebase, a set of PDFs, and a Notion workspace — and route each question to the right index automatically. LlamaIndex’s RouterQueryEngine does exactly this. This tutorial builds a working multi-document RAG pipeline from scratch, showing how to configure retrievers, define routing logic, and handle the edge cases that appear in production.

Prerequisites

Python 3.10+
An OpenAI API key (or any LlamaIndex-compatible LLM/embedding provider)
Basic familiarity with LlamaIndex concepts: documents, nodes, indexes, query engines

Install the required packages:

pip install llama-index llama-index-llms-openai llama-index-embeddings-openai \
    llama-index-readers-file pypdf python-dotenv

Create a .env file with your API key:

OPENAI_API_KEY=sk-...

Project structure

multi-doc-rag/
├── data/
│   ├── engineering/
│   │   ├── api-spec.md
│   │   └── database-schema.sql
│   ├── product/
│   │   ├── roadmap-q3.md
│   │   └── user-feedback.md
│   └── legal/
│       ├── privacy-policy.md
│       └── terms-of-service.md
├── main.py
└── .env

Populate the data/ directories with real files. The examples below assume markdown and SQL files, but any format LlamaIndex readers support will work.

Load and index each collection separately

The key insight: each domain gets its own index with its own retriever configuration. This lets you tune chunk size, similarity cutoff, and metadata filters per collection.

# main.py
import os
from pathlib import Path
from dotenv import load_dotenv

from llama_index.core import (
    SimpleDirectoryReader,
    VectorStoreIndex,
    StorageContext,
    Settings,
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

load_dotenv()

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=64)

DATA_ROOT = Path("data")
COLLECTIONS = ["engineering", "product", "legal"]

def build_index(collection_name: str) -> VectorStoreIndex:
    """Load documents from data/<collection> and build a vector index."""
    docs = SimpleDirectoryReader(
        input_dir=DATA_ROOT / collection_name,
        recursive=True,
        required_exts=[".md", ".sql", ".txt", ".pdf"],
    ).load_data()
    
    print(f"Loaded {len(docs)} documents from {collection_name}")
    for d in docs:
        print(f"  - {d.metadata.get('file_name', 'unknown')}")
    
    index = VectorStoreIndex.from_documents(docs, show_progress=True)
    return index

indices = {name: build_index(name) for name in COLLECTIONS}

Run this and verify output:

Loaded 2 documents from engineering
  - api-spec.md
  - database-schema.sql
Loaded 2 documents from product
  - roadmap-q3.md
  - user-feedback.md
Loaded 2 documents from legal
  - privacy-policy.md
  - terms-of-service.md

Create retrievers with domain-specific configuration

Different collections need different retrieval strategies. Engineering docs benefit from tighter similarity thresholds; legal docs may need broader recall.

from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.postprocessor import SimilarityPostprocessor

retrievers = {}
for name, index in indices.items():
    if name == "engineering":
        # Precise retrieval for technical specs
        retriever = VectorIndexRetriever(
            index=index,
            similarity_top_k=4,
        )
        retriever = SimilarityPostprocessor(retriever, similarity_cutoff=0.75)
    elif name == "legal":
        # Broader recall for policy questions
        retriever = VectorIndexRetriever(
            index=index,
            similarity_top_k=6,
        )
        retriever = SimilarityPostprocessor(retriever, similarity_cutoff=0.65)
    else:
        # Default balanced config
        retriever = VectorIndexRetriever(
            index=index,
            similarity_top_k=5,
        )
        retriever = SimilarityPostprocessor(retriever, similarity_cutoff=0.70)
    
    retrievers[name] = retriever

Define the routing logic

RouterQueryEngine needs a selector that maps queries to retrievers. LlamaIndex provides LLMSingleSelector and LLMMultiSelector — the former picks one index, the latter can combine multiple. For most multi-document RAG, start with single selection.

from llama_index.core.query_engine import RouterQueryEngine
from llama_index.core.selectors import LLMSingleSelector
from llama_index.core.tools import QueryEngineTool
from llama_index.core import PromptTemplate

# Build a query engine per collection
query_engines = {}
for name, retriever in retrievers.items():
    query_engines[name] = indices[name].as_query_engine(
        retriever=retriever,
        response_mode="compact",
    )

# Wrap each as a tool with a clear description for the selector LLM
tools = [
    QueryEngineTool.from_defaults(
        query_engine=query_engines["engineering"],
        name="engineering_docs",
        description=(
            "Technical documentation: API specifications, database schemas, "
            "infrastructure configs, and codebase architecture. "
            "Use for questions about endpoints, data models, deployment, "
            "or system internals."
        ),
    ),
    QueryEngineTool.from_defaults(
        query_engine=query_engines["product"],
        name="product_docs",
        description=(
            "Product planning and user feedback: roadmaps, feature specs, "
            "user research, and prioritization decisions. "
            "Use for questions about upcoming features, product strategy, "
            "or customer requests."
        ),
    ),
    QueryEngineTool.from_defaults(
        query_engine=query_engines["legal"],
        name="legal_docs",
        description=(
            "Legal and compliance documents: privacy policy, terms of service, "
            "data processing agreements. "
            "Use for questions about data handling, user rights, "
            "or regulatory obligations."
        ),
    ),
]

# The selector uses these descriptions to route
selector = LLMSingleSelector.from_defaults(
    llm=Settings.llm,
    verbose=True,
)

router_engine = RouterQueryEngine(
    selector=selector,
    query_engine_tools=tools,
    verbose=True,
)

The verbose=True flags print the selector’s reasoning — essential for debugging routing decisions.

Test the router with representative queries

test_queries = [
    "What are the required fields for the /v1/users endpoint?",
    "When is the dark mode feature scheduled for release?",
    "What personal data do we collect under the privacy policy?",
    "How do we handle database migrations in production?",
    "What are the top user complaints about onboarding?",
]

for q in test_queries:
    print(f"\n{'='*60}")
    print(f"QUERY: {q}")
    print(f"{'='*60}")
    response = router_engine.query(q)
    print(f"RESPONSE: {response}")
    print(f"SOURCE NODES: {len(response.source_nodes)}")
    for node in response.source_nodes:
        print(f"  - {node.metadata.get('file_name', 'unknown')} (score: {node.score:.3f})")

Expected output pattern:

============================================================
QUERY: What are the required fields for the /v1/users endpoint?
============================================================
Selecting query engine: engineering_docs
Reasoning: The query asks about API endpoint specifications, which falls under technical documentation...
RESPONSE: The /v1/users endpoint requires the following fields: email (string, format: email), 
  name (string, min 2 chars), and role (enum: admin, member, viewer). Optional fields include 
  avatar_url and timezone.
SOURCE NODES: 3
  - api-spec.md (score: 0.872)
  - api-spec.md (score: 0.841)
  - database-schema.sql (score: 0.793)

============================================================
QUERY: When is the dark mode feature scheduled for release?
============================================================
Selecting query engine: product_docs
Reasoning: The query asks about a feature release timeline, which is product planning...
RESPONSE: Dark mode is scheduled for Q3 2024, targeting the week of August 19th. 
  It's currently in design review with engineering implementation starting July 15th.
SOURCE NODES: 2
  - roadmap-q3.md (score: 0.912)
  - user-feedback.md (score: 0.687)

Handle multi-collection queries with LLMMultiSelector

Some questions genuinely need multiple sources. Switch the selector:

from llama_index.core.selectors import LLMMultiSelector

multi_selector = LLMMultiSelector.from_defaults(
    llm=Settings.llm,
    verbose=True,
    max_outputs=3,  # cap at 3 collections
)

multi_router = RouterQueryEngine(
    selector=multi_selector,
    query_engine_tools=tools,
    verbose=True,
    # Combine responses from multiple engines
    response_mode="compact",
)

Test a cross-cutting query:

cross_query = "What data protection measures apply to user data in the new analytics feature?"
response = multi_router.query(cross_query)

Output shows multiple engines selected:

Selecting query engines: ['product_docs', 'legal_docs', 'engineering_docs']
Reasoning: This question spans product feature details, legal compliance requirements, 
  and technical implementation of data protection...
RESPONSE: The analytics feature (product_docs) implements pseudonymization at ingestion 
  (engineering_docs) and retains data for 13 months per GDPR Article 5 (legal_docs). 
  Access is restricted to the analytics service account with audit logging enabled.
SOURCE NODES: 5
  - roadmap-q3.md (score: 0.834)
  - privacy-policy.md (score: 0.791)
  - api-spec.md (score: 0.756)
  - terms-of-service.md (score: 0.712)
  - database-schema.sql (score: 0.689)

Persist indices to avoid re-indexing

Re-building indices on every startup wastes time and API calls. Persist to disk:

PERSIST_DIR = Path("storage")

def persist_indices(indices: dict, base_dir: Path):
    base_dir.mkdir(exist_ok=True)
    for name, index in indices.items():
        index.storage_context.persist(persist_dir=base_dir / name)
    print(f"Persisted {len(indices)} indices to {base_dir}")

def load_indices(base_dir: Path) -> dict:
    from llama_index.core import load_index_from_storage
    loaded = {}
    for name in COLLECTIONS:
        storage_context = StorageContext.from_defaults(persist_dir=base_dir / name)
        loaded[name] = load_index_from_storage(storage_context)
    return loaded

# After building:
persist_indices(indices, PERSIST_DIR)

# On subsequent runs:
# indices = load_indices(PERSIST_DIR)

Add metadata filtering for tighter control

When documents share a collection but differ in version, team, or sensitivity, attach metadata at ingestion and filter at query time.

from llama_index.core.schema import Document
from llama_index.core.vector_stores import MetadataFilters, ExactMatchFilter

# Example: tag engineering docs with version during ingestion
def load_with_metadata(collection: str) -> list[Document]:
    docs = SimpleDirectoryReader(
        input_dir=DATA_ROOT / collection,
        recursive=True,
    ).load_data()
    
    for doc in docs:
        doc.metadata["collection"] = collection
        # Extract version from filename or content
        if "v2" in doc.metadata.get("file_name", ""):
            doc.metadata["version"] = "v2"
        else:
            doc.metadata["version"] = "v1"
    return docs

# At query time, filter by metadata
from llama_index.core.retrievers import VectorIndexRetriever

versioned_retriever = VectorIndexRetriever(
    index=indices["engineering"],
    similarity_top_k=4,
    filters=MetadataFilters(
        filters=[ExactMatchFilter(key="version", value="v2")]
    ),
)

Common failure modes and fixes

Symptom	Cause	Fix
Router always picks the same tool	Tool descriptions too similar	Rewrite descriptions with distinct keywords; add negative examples
Selector hallucinates a tool name	LLM doesn’t respect tool list	Use `PydanticSingleSelector` for structured output; lower temperature
Empty results from selected engine	Similarity cutoff too high	Lower `similarity_cutoff` or increase `similarity_top_k`
Multi-selector returns all tools	`max_outputs` too high or query too vague	Set `max_outputs=2`; add a “general” catch-all tool with low priority
Slow latency	Multiple sequential LLM calls	Use `n4n.ai` or similar gateway for parallel provider routing and cached embeddings

Production considerations

Observability: Wrap the router with a callback handler to log selection reasoning, latency per engine, and token usage.

from llama_index.core.callbacks import CallbackManager, LlamaDebugHandler

debug_handler = LlamaDebugHandler(print_trace_on_end=True)
Settings.callback_manager = CallbackManager([debug_handler])

Evaluation: Build a small eval set (20-30 queries with expected tool selections) and measure routing accuracy before deploying changes to descriptions or thresholds.

Fallback: Add a default vector index over all documents as a catch-all tool with the lowest priority description. This handles queries that don’t cleanly map to any domain.

# Catch-all index over everything
all_docs = []
for name in COLLECTIONS:
    all_docs.extend(SimpleDirectoryReader(input_dir=DATA_ROOT / name).load_data())

catchall_index = VectorStoreIndex.from_documents(all_docs)
catchall_tool = QueryEngineTool.from_defaults(
    query_engine=catchall_index.as_query_engine(similarity_top_k=5),
    name="general_docs",
    description="General company knowledge. Use only when no other tool matches.",
)
tools.append(catchall_tool)

Summary

You now have a multi-document RAG system that:

Maintains separate indices per domain with tuned retrieval
Routes queries automatically via LLM-based selection
Supports both single and multi-collection queries
Persists indices for fast startup
Includes metadata filtering for versioned content
Has observability hooks for production debugging

The router pattern scales well to 10-20 collections. Beyond that, consider hierarchical routing (category → sub-category) or a dedicated classifier model instead of LLM selection.