Building a multi-document summary index in LlamaIndex

Most RAG pipelines flatten everything into one vector store and hope the retriever guesses right. This llamaindex document summary index tutorial shows how to build a multi-document summary index that summarizes each source up front, then routes queries to the few docs that actually matter. You get coarse-grained retrieval first, fine-grained answer synthesis second.

Prerequisites

• Python 3.10 or newer
• llama-index pinned to a 0.10.x release (tested on 0.10.20)
• An OpenAI API key, or any OpenAI-compatible endpoint
• Familiarity with basic LlamaIndex Document and QueryEngine concepts

pip install llama-index==0.10.20
export OPENAI_API_KEY="sk-..."

If you run this in a notebook, restart the kernel after install.

Step 1: Create sample documents

We’ll use three short documents with distinct topics. In practice these would be PDFs, Notion pages, or support tickets.

from llama_index import Document

docs = [
    Document(
        text="The API supports OAuth2 and API key auth. OAuth2 tokens expire after 3600 seconds. "
             "Refresh tokens are valid for 30 days. Rate limits are 100 req/min on the free tier.",
        metadata={"source": "auth_docs", "team": "platform"},
    ),
    Document(
        text="The billing system uses Stripe webhooks. Invoices are finalized on the 1st of each month. "
             "Failed charges retry three times with exponential backoff. Customers can export CSV receipts.",
        metadata={"source": "billing_docs", "team": "finance"},
    ),
    Document(
        text="The search service indexes documents asynchronously. New uploads appear within 60 seconds. "
             "Semantic search uses embeddings with cosine similarity. Filtering by tag is exact match.",
        metadata={"source": "search_docs", "team": "search"},
    ),
]

Step 2: Configure the LLM

DocumentSummaryIndex needs an LLM to write the per-document summaries. The default OpenAI class works out of the box.

from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-3.5-turbo", temperature=0)

If you want a single OpenAI-compatible endpoint that fronts 240+ models with automatic fallback when a provider is degraded, point the client at n4n.ai instead of OpenAI directly:

llm = OpenAI(
    model="openai/gpt-4o-mini",
    api_base="https://api.n4n.ai/v1",
    api_key="YOUR_N4N_KEY",
    temperature=0,
)

Either works for the rest of this tutorial.

Step 3: Build the summary index

The index constructor calls the LLM once per document to produce a concise summary, then embeds those summaries into an internal vector store.

from llama_index.indices.document_summary import DocumentSummaryIndex

index = DocumentSummaryIndex.from_documents(
    docs,
    llm=llm,
    show_progress=True,
)

Expected console output (abridged):

Generating summary for document 0...
Generating summary for document 1...
Generating summary for document 2...
Index built with 3 documents.

Under the hood, each summary is stored as a node. The original documents remain in the docstore for later drill-down.

Step 4: Inspect per-document summaries

Before querying, verify the summaries capture the right signal.

doc_ids = list(index.docstore.get_document_hash().keys())
for doc_id in doc_ids:
    print(f"--- {doc_id} ---")
    print(index.get_document_summary(doc_id))
    print()

Sample output:

--- 4f3c... ---
The document describes API authentication using OAuth2 and API keys, token expiry and refresh windows, and free-tier rate limits.

--- 9a12... ---
The text explains Stripe-based billing, monthly invoice finalization, retry logic for failed charges, and CSV receipt exports.

--- b7e0... ---
Covers async search indexing, 60-second freshness lag, cosine similarity for semantic search, and exact-match tag filters.

If a summary is too verbose or misses a field you care about, customize the prompt (see Step 6).

Step 5: Query across documents

The query engine first retrieves the most relevant document summaries, then pulls the full text of those documents to synthesize a final answer.

query_engine = index.as_query_engine()
response = query_engine.query(
    "What auth methods are available and what are the rate limits?"
)
print(str(response))

Expected answer:

The API supports OAuth2 and API key authentication. OAuth2 tokens expire after 3600 seconds and refresh tokens last 30 days. On the free tier, rate limits are 100 requests per minute.

Notice the response only used the auth_docs document. The other two never entered the context window. That’s the core win of the llamaindex document summary index tutorial pattern: you skip irrelevant docs entirely.

Step 6: Customize the summary prompt

The default summary prompt is generic. For compliance or support use cases you often want a fixed schema.

from llama_index.core.prompts import PromptTemplate

summary_template = PromptTemplate(
    "Summarize the document in 2 sentences. "
    "Explicitly list any numbers, time windows, or limits. "
    "Document:\n{context_str}\nSummary:"
)

index = DocumentSummaryIndex.from_documents(
    docs,
    llm=llm,
    summary_kwargs={"summary_template": summary_template},
    show_progress=True,
)

Re-run Step 4 and you’ll see tighter summaries that surface metrics explicitly. This matters when the query engine later decides which docs to expand.

Step 7: Persist and reload

Re-summarizing on every boot wastes tokens. Persist the index to disk.

index.storage_context.persist(persist_dir="./summary_index")

Reload it later:

from llama_index.core.storage_storage_context import StorageContext
from llama_index.indices.document_summary import DocumentSummaryIndex

storage_context = StorageContext.from_defaults(persist_dir="./summary_index")
index = DocumentSummaryIndex.load_from_storage(storage_context, llm=llm)

The summaries and vector embeddings are restored; only query-time LLM calls hit the network.

Step 8: Route by metadata before query

A multi-document summary index still scans all summaries. If you have thousands of docs, pre-filter by metadata using a VectorStoreIndex hybrid or a simple MetadataFilter.

from llama_index.core.vector_stores import MetadataFilters, ExactMatchFilter

filtered_engine = index.as_query_engine(
    filters=MetadataFilters(
        filters=[ExactMatchFilter(key="team", value="platform")]
    )
)

This restricts summary retrieval to the platform team’s docs, cutting latency and cost.

Production notes

• Summary cost is upfront, not per-query. With 1k docs and a 200-token summary each, you pay for ~200k output tokens once. Queries then only spend tokens on the 1–3 docs retrieved.
• Chunk size still matters. DocumentSummaryIndex stores the full document for drill-down. If a single document is a 200-page PDF, wrap it in a SentenceSplitter before indexing, or the drill-down context will blow up.
• Embedding model choice. Summaries are short; a small embedding model (e.g., text-embedding-3-small) is usually sufficient. Swap it via embed_model in Settings.
• Cache summaries. Because n4n.ai and other gateways forward provider cache-control hints, you can mark the summary generation calls as cacheable to avoid re-paying if you rebuild the index with the same doc text.

The llamaindex document summary index tutorial above gives you a working multi-document retrieval layer that scales better than dumping everything into one vector search. Start with three docs, confirm the summaries, then expand.