LlamaIndex document summary index for large corpora

The LlamaIndex document summary index large corpora tutorial you’re reading exists because most engineers reach for a vector index first, hit latency or cost walls at scale, and only then realize a summary index better fits their access pattern. A document summary index builds a hierarchical summary tree over your corpus, letting you answer broad questions without retrieving every chunk. This guide walks through configuring it for millions of tokens, choosing the right summarization strategy, and avoiding the memory and latency traps that appear in production.

When a summary index beats a vector index

Vector indexes excel at “find me the paragraph about X.” Summary indexes excel at “what does this corpus say about X?” If your queries are broad — competitive analysis across 500 earnings calls, theme extraction from 10,000 support tickets, or “summarize the last quarter of engineering RFCs” — a vector index forces you to retrieve and rerank hundreds of chunks. A summary index answers from pre-computed summaries, often in one or two LLM calls.

The tradeoff: you lose fine-grained citation. A summary index tells you what the corpus says, not where it says it. If you need page-level citations, keep a vector index alongside and route queries accordingly.

Build the index incrementally

Loading 50,000 documents into memory at once OOMs most instances. Use SimpleDirectoryReader with filename_as_id=True and stream documents in batches. Persist after each batch so a crash doesn’t lose progress.

from llama_index.core import (
    DocumentSummaryIndex,
    SimpleDirectoryReader,
    StorageContext,
    load_index_from_storage,
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o-mini", temperature=0)
splitter = SentenceSplitter(chunk_size=1024, chunk_overlap=128)

storage_context = StorageContext.from_defaults(persist_dir="./summary_index")
index = None

reader = SimpleDirectoryReader(
    input_dir="./corpus",
    filename_as_id=True,
    recursive=True,
)

for batch in reader.iter_data(batch_size=100):
    documents = []
    for doc in batch:
        nodes = splitter.get_nodes_from_documents([doc])
        documents.extend(nodes)

    if index is None:
        index = DocumentSummaryIndex.from_documents(
            documents,
            llm=llm,
            storage_context=storage_context,
            show_progress=True,
        )
    else:
        index.insert_nodes(documents)

    index.storage_context.persist()

Key settings: chunk_size=1024 keeps leaf summaries focused. Larger chunks produce vague summaries; smaller chunks explode the tree depth. gpt-4o-mini balances cost and instruction following for summarization. If you run on n4n.ai, you can swap the LLM to any of 240+ models without changing client code — useful when you need a larger context window for the final synthesis step.

Choose the right response mode

DocumentSummaryIndex supports three response modes. Pick one per query, not globally.

from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.retrievers import DocumentSummaryIndexRetriever

retriever = DocumentSummaryIndexRetriever(
    index=index,
    choice_batch_size=10,      # how many summaries to consider per level
    choice_top_k=3,            # how many to keep per level
    mode="tree_summarize",
)

query_engine = RetrieverQueryEngine.from_args(
    retriever,
    response_mode="tree_summarize",
    llm=llm,
)

response = query_engine.query(
    "What are the top three risks mentioned across all 2024 board decks?"
)

choice_batch_size and choice_top_k control the breadth-depth tradeoff. Higher values improve recall but increase latency and token spend. Start with 10/3 and tune after measuring.

Handle large corpora with async and caching

Synchronous summarization of 100,000 documents takes hours. Use async insertion and enable the built-in summary cache.

import asyncio
from llama_index.core import DocumentSummaryIndex

async def build_index_async(documents):
    index = await DocumentSummaryIndex.afrom_documents(
        documents,
        llm=llm,
        storage_context=storage_context,
        show_progress=True,
        insert_batch_size=50,        # parallelize across batches
        summary_query="Summarize this document in 3 sentences focusing on decisions, risks, and metrics.",
    )
    return index

# Run in your async entrypoint
index = asyncio.run(build_index_async(all_documents))

The summary_query parameter is your lever for domain-specific summaries. Default is generic; a tailored prompt cuts token usage 30-50% by avoiding irrelevant detail.

# Domain-tuned summary prompt for earnings calls
EARNINGS_SUMMARY_QUERY = (
    "Summarize this earnings call transcript in 4 sentences. "
    "Extract: (1) revenue and EPS vs expectations, "
    "(2) forward guidance changes, "
    "(3) margin drivers or headwinds, "
    "(4) capital allocation signals (buybacks, M&A, capex)."
)

Pitfall: summary drift at depth

A tree summary index compresses at each level. By level 3-4, specific numbers, names, and dates vanish. If your queries need precise figures (“what was Q3 revenue?”), the summary index will hallucinate or say “not mentioned.”

Mitigations:

• Keep chunk_size small enough that leaf summaries retain key facts
• Set num_children=10 (default) to limit tree height; more children = shallower tree
• For fact-seeking queries, fall back to a vector index with similarity_top_k=20 and a reranker

from llama_index.core.query_engine import RouterQueryEngine
from llama_index.core.selectors import LLMSingleSelector
from llama_index.core.tools import QueryEngineTool

summary_tool = QueryEngineTool.from_defaults(
    query_engine=summary_query_engine,
    description="Use for broad thematic questions across the entire corpus.",
)
vector_tool = QueryEngineTool.from_defaults(
    query_engine=vector_query_engine,
    description="Use for specific fact lookup, citations, or narrow document queries.",
)

router = RouterQueryEngine(
    selector=LLMSingleSelector.from_defaults(llm=llm),
    query_engine_tools=[summary_tool, vector_tool],
)

The router adds one LLM call per query but eliminates the “wrong tool” failure mode.

Pitfall: insertion latency scales linearly

Each inserted document triggers a summary call. At 50k documents, that’s 50k LLM calls. Batch insertion helps, but you still pay per document. Two strategies:

1. Pre-summarize offline: Run a cheap model (Llama-3.1-8B, Gemma-2-9B) over raw documents, store summaries as metadata, then build the index from pre-summarized nodes. Cuts cost 10x.
2. Incremental rebuilds: Don’t rebuild nightly. Insert new documents daily; rebuild the full tree weekly. The index supports insert_nodes and delete_ref_doc for true incremental updates.

# Daily incremental insert
new_docs = load_new_documents(since=last_run)
new_nodes = splitter.get_nodes_from_documents(new_docs)
index.insert_nodes(new_nodes)
index.storage_context.persist()

# Weekly full rebuild (cron job)
# Rebuild from scratch to rebalance tree and refresh stale summaries
index = DocumentSummaryIndex.from_documents(
    all_nodes,
    llm=llm,
    storage_context=storage_context,
)

Monitor what matters

Log these metrics per query to catch degradation early:

import time
from llama_index.core.callbacks import CallbackManager, TokenCountingHandler

token_counter = TokenCountingHandler()
callback_manager = CallbackManager([token_counter])

# Attach to your query engine
query_engine.callback_manager = callback_manager

start = time.perf_counter()
response = query_engine.query("...")
latency_ms = (time.perf_counter() - start) * 1000

print(f"latency_ms={latency_ms:.0f} "
      f"prompt_tokens={token_counter.prompt_llm_token_count} "
      f"completion_tokens={token_counter.completion_llm_token_count} "
      f"total_tokens={token_counter.total_llm_token_count}")

Alert on:

• latency_ms > p99 baseline (tree_summarize is inherently variable)
• total_tokens growing without query volume increase (summary bloat)
• prompt_tokens / completion_tokens > 20 (context stuffing, not summarizing)

Production checklist

Before shipping:

• Persistence verified: Kill the process mid-build, restart, confirm index loads and queries
• Memory profile: Run memory_profiler on a 10k-doc build; ensure peak RSS fits your container limit
• Cost model: Estimate doc_count * avg_summary_tokens * $/1k for build + daily_queries * avg_query_tokens for serving
• Fallback tested: Simulate provider outage; confirm router falls back to vector index gracefully
• Stale summary policy: Define TTL for summaries (e.g., rebuild weekly for financial docs, monthly for policy docs)

Summary index vs. knowledge graph index

If your queries are relational — “which executives appear across both the risk committee and compensation committee docs?” — a summary index cannot answer. Knowledge graph indexes extract entities and relationships, enabling graph traversal. They cost more to build (entity extraction + triple extraction per chunk) but unlock a different query class. Use both: summary index for thematic queries, knowledge graph for relational queries, vector index for fact lookup.

Closing thought

The document summary index is a specialized tool. It shines when query scope is broad and citation granularity is low. Configure it with domain-tuned summary prompts, incremental builds, and a router in front. Monitor token spend per query like you monitor latency — both are costs you control.