LlamaIndex ComposableGraph for multi-document indexes

LlamaIndex ComposableGraph lets you treat multiple document indexes as a single queryable structure. Instead of merging everything into one giant index — which loses document boundaries and makes routing impossible — you compose smaller indexes into a graph where each node represents a distinct data source. This tutorial walks through building a multi-document RAG system that routes queries to the right index automatically.

Prerequisites

You need Python 3.10+ and an OpenAI-compatible API key. Install the core packages:

pip install llama-index llama-index-llms-openai llama-index-embeddings-openai

If you’re running against a local model or a gateway like n4n.ai, swap the base URL in the LLM and embedding constructors — the rest of the code stays the same.

Create a project directory and add two sample document sets. For this tutorial we’ll use a pair of markdown files representing separate knowledge domains:

mkdir -p data/engineering data/product

data/engineering/architecture.md:

# System Architecture

The payment service uses an event-driven architecture with Kafka as the message backbone.
Services communicate via Avro schemas registered in Confluent Schema Registry.

## Database Layer
- PostgreSQL for transactional data (orders, users, accounts)
- Redis for session caching and rate limiting counters
- ClickHouse for analytical queries on payment events

## Deployment
Kubernetes on GKE with Istio for mTLS and traffic splitting.
ArgoCD manages GitOps deployments.

data/product/pricing.md:

# Pricing Model

## Transaction Fees
- Domestic cards: 2.9% + $0.30 per successful charge
- International cards: 3.9% + $0.30
- ACH transfers: 0.8% capped at $5.00

## Volume Discounts
- $100K–$500K/mo: 2.7% + $0.30
- $500K–$1M/mo: 2.5% + $0.25
- $1M+/mo: custom pricing

## Dispute Fees
$15 per disputed charge, refunded if resolved in merchant's favor.

Step 1: Load documents into separate indexes

Each domain gets its own vector index. This preserves semantic boundaries and lets the graph router decide which index to query.

# step1_build_indexes.py
from pathlib import Path
from llama_index.core import (
    SimpleDirectoryReader,
    VectorStoreIndex,
    StorageContext,
    Settings,
)
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI

# Configure global settings — swap base_url for your gateway
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

def build_index(doc_dir: str, persist_dir: str) -> VectorStoreIndex:
    documents = SimpleDirectoryReader(doc_dir).load_data()
    index = VectorStoreIndex.from_documents(documents)
    index.storage_context.persist(persist_dir=persist_dir)
    return index

if __name__ == "__main__":
    eng_index = build_index("data/engineering", "storage/engineering")
    prod_index = build_index("data/product", "storage/product")
    print("Indexes built and persisted.")

Run it:

python step1_build_indexes.py

Expected output:

Indexes built and persisted.

Two directories now exist under storage/ with their own vector stores and docstores.

Step 2: Define index summaries for routing

ComposableGraph needs a summary for each index so the LLM can route queries. The summary should describe what questions this index answers, not just what documents it contains.

# step2_summaries.py
from llama_index.core import load_index_from_storage, StorageContext

ENG_SUMMARY = (
    "Contains technical architecture details for the payment platform: "
    "service topology, database choices, messaging infrastructure, "
    "deployment model, and security configuration. "
    "Use for questions about system design, tech stack, or operations."
)

PROD_SUMMARY = (
    "Contains pricing, fees, and commercial terms for payment processing: "
    "transaction rates, volume discounts, dispute fees, and regional pricing. "
    "Use for questions about costs, billing, or fee schedules."
)

def load_index(persist_dir: str):
    storage = StorageContext.from_defaults(persist_dir=persist_dir)
    return load_index_from_storage(storage)

if __name__ == "__main__":
    eng = load_index("storage/engineering")
    prod = load_index("storage/product")
    print("Engineering index loaded:", len(eng.docstore.docs), "nodes")
    print("Product index loaded:", len(prod.docstore.docs), "nodes")

Step 3: Build the ComposableGraph

Now compose the indexes into a graph. We’ll use the ComposableGraph factory with a SummaryIndex as the root — this lets the graph route by comparing the query against each index summary.

# step3_build_graph.py
from llama_index.core import ComposableGraph, SummaryIndex
from llama_index.core.indices.composability import GraphRootType
from step2_summaries import load_index, ENG_SUMMARY, PROD_SUMMARY

def build_graph():
    eng_index = load_index("storage/engineering")
    prod_index = load_index("storage/product")

    graph = ComposableGraph.from_indices(
        SummaryIndex,
        [eng_index, prod_index],
        index_summaries=[ENG_SUMMARY, PROD_SUMMARY],
        root_kwargs={"index_struct_type": GraphRootType.DEFAULT},
    )
    graph.storage_context.persist("storage/graph")
    return graph

if __name__ == "__main__":
    g = build_graph()
    print("Graph built with", len(g.all_indices), "child indexes")
    print("Root index type:", type(g.root_index).__name__)

Run it:

python step3_build_graph.py

Output:

Graph built with 2 child indexes
Root index type: SummaryIndex

The graph now has a root SummaryIndex that holds the two summaries and knows which child index each summary maps to.

Step 4: Query the graph

The graph exposes a query_engine that handles routing automatically. The root index selects the relevant child index(es), then the child index retrieves and synthesizes an answer.

# step4_query.py
from llama_index.core import ComposableGraph, StorageContext

def load_graph():
    storage = StorageContext.from_defaults(persist_dir="storage/graph")
    return ComposableGraph.load_from_storage(storage)

def main():
    graph = load_graph()
    query_engine = graph.as_query_engine(
        similarity_top_k=3,
        response_mode="compact",
    )

    questions = [
        "What message backbone does the payment service use?",
        "What's the fee for a domestic card transaction?",
        "How are services deployed?",
        "What's the dispute fee and when is it refunded?",
        "Which database handles analytical queries?",
    ]

    for q in questions:
        print(f"\n>>> {q}")
        response = query_engine.query(q)
        print(response.response)
        print(f"[Source nodes: {len(response.source_nodes)}]")
        for node in response.source_nodes:
            print(f"  - {node.metadata.get('file_path', 'unknown')} (score: {node.score:.3f})")

if __name__ == "__main__":
    main()

Run it:

python step4_query.py

Expected output (abridged):

>>> What message backbone does the payment service use?
The payment service uses Kafka as its message backbone.
[Source nodes: 2]
  - data/engineering/architecture.md (score: 0.892)
  - data/engineering/architecture.md (score: 0.841)

>>> What's the fee for a domestic card transaction?
Domestic card transactions incur a fee of 2.9% + $0.30 per successful charge.
[Source nodes: 2]
  - data/product/pricing.md (score: 0.915)
  - data/product/pricing.md (score: 0.873)

>>> How are services deployed?
Services are deployed on Kubernetes (GKE) with Istio for mTLS and traffic splitting, managed via ArgoCD GitOps.
[Source nodes: 2]
  - data/engineering/architecture.md (score: 0.888)
  - data/engineering/architecture.md (score: 0.834)

>>> What's the dispute fee and when is it refunded?
The dispute fee is $15 per disputed charge, refunded if the dispute is resolved in the merchant's favor.
[Source nodes: 1]
  - data/product/pricing.md (score: 0.901)

>>> Which database handles analytical queries?
ClickHouse handles analytical queries on payment events.
[Source nodes: 1]
  - data/engineering/architecture.md (score: 0.879)

The router correctly sends architecture questions to the engineering index and pricing questions to the product index.

Step 5: Custom routing with a keyword index

Summary-based routing works well for semantic separation. For exact-match or keyword-heavy domains (error codes, SKUs, API endpoints), add a KeywordTableIndex as a child and give it a summary that emphasizes exact terminology.

# step5_keyword_child.py
from llama_index.core import (
    ComposableGraph,
    KeywordTableIndex,
    SimpleDirectoryReader,
    StorageContext,
    Settings,
)
from llama_index.core.indices.composability import GraphRootType
from step2_summaries import load_index, ENG_SUMMARY, PROD_SUMMARY

ERROR_SUMMARY = (
    "Contains error codes, exception messages, and troubleshooting steps "
    "for the payment platform. Use when the query includes specific error codes "
    "like 'PAY-402', 'KAFKA-503', or exact exception names."
)

def build_with_keyword():
    # Load existing indexes
    eng_index = load_index("storage/engineering")
    prod_index = load_index("storage/product")

    # Build a new keyword index from a third doc set
    error_docs = SimpleDirectoryReader("data/errors").load_data()
    error_index = KeywordTableIndex.from_documents(error_docs)
    error_index.storage_context.persist("storage/errors")

    # Compose all three
    graph = ComposableGraph.from_indices(
        SummaryIndex,
        [eng_index, prod_index, error_index],
        index_summaries=[ENG_SUMMARY, PROD_SUMMARY, ERROR_SUMMARY],
        root_kwargs={"index_struct_type": GraphRootType.DEFAULT},
    )
    graph.storage_context.persist("storage/graph_v2")
    return graph

Create data/errors/troubleshooting.md with content like:

# Error Codes

## PAY-402: Insufficient Funds
Triggered when the funding source has insufficient balance.
Resolution: Retry with a different payment method.

## KAFKA-503: Broker Unavailable
Producer cannot reach any Kafka broker.
Resolution: Check Istio sidecar health and Kafka cluster status.

Now queries containing “PAY-402” or “KAFKA-503” route to the keyword index, which excels at exact term matching.

Step 6: Persist and reload in production

In production you build the graph once (CI/CD, nightly job, or on deploy) and load it at service startup. The ComposableGraph.load_from_storage method reconstructs the entire structure — root index, child indexes, and the summary-to-child mapping — from the persisted directories.

# production_loader.py
from llama_index.core import ComposableGraph, StorageContext
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core import Settings

Settings.llm = OpenAI(model="gpt-4o-mini")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

def get_query_engine(graph_dir: str = "storage/graph"):
    storage = StorageContext.from_defaults(persist_dir=graph_dir)
    graph = ComposableGraph.load_from_storage(storage)
    return graph.as_query_engine(similarity_top_k=4)

# In your FastAPI/Flask handler:
# engine = get_query_engine()
# response = engine.query(user_question)

When to use ComposableGraph vs. a single index

Common pitfalls

Overlapping summaries — If two summaries describe similar content, the router picks arbitrarily. Make summaries mutually exclusive by focusing on question types, not just topics.

Too many children — The root SummaryIndex embeds all summaries in a single prompt. Beyond ~10–15 children, routing degrades. For larger fleets, build a two-level graph: domain graphs at the first level, a meta-graph at the root.

Ignoring metadata — Child indexes preserve document metadata. Use metadata_filters on the query engine when you need tenant isolation or version scoping:

from llama_index.core.vector_stores import MetadataFilters, ExactMatchFilter

engine = graph.as_query_engine(
    filters=MetadataFilters(filters=[ExactMatchFilter(key="tenant_id", value="acme")])
)

Stale persisted state — If you change the embedding model, rebuild all child indexes and the graph. The root index stores summary embeddings that must match the child index embedding space.

Next steps

• Add a TreeIndex child for hierarchical docs (API specs with nested endpoints)
• Implement a custom GraphRootRetriever that logs routing decisions for observability
• Use ComposableGraph.from_indices with root_index_cls=VectorStoreIndex for dense retrieval at the root level instead of summary matching

The graph abstraction keeps your retrieval logic honest: each index does one thing well, and the router decides which one gets the query. That scales better than stuffing everything into a single vector space and hoping similarity search figures it out.