LlamaIndex tree summarize vs refine response mode

If you’re building a RAG pipeline with LlamaIndex, the response mode you choose — tree_summarize or refine — directly controls latency, token spend, and answer quality. This comparison breaks down the mechanics, trade-offs, and decision criteria so you can pick the right one without guessing.

How each mode works

Refine: sequential iteration

Refine processes retrieved nodes one at a time. It sends the first node plus your query to the LLM, gets an initial answer, then feeds that answer plus the next node back to the LLM to refine it. This repeats until all nodes are consumed or the context window fills.

from llama_index.core import VectorStoreIndex
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.response_synthesizers import Refine

index = VectorStoreIndex.from_documents(docs)
query_engine = index.as_query_engine(
    response_mode="refine",
    # or explicitly:
    # response_synthesizer=Refine(llm=llm)
)
response = query_engine.query("What were the key findings?")

Each iteration sees the accumulated answer so far. The LLM can correct, expand, or contradict earlier reasoning as new evidence arrives. This is powerful for complex synthesis but inherently sequential — you can’t parallelize across nodes.

Tree summarize: bottom-up reduction

Tree_summarize builds a summary tree. It groups nodes into batches that fit the context window, summarizes each batch in parallel, then recursively summarizes those summaries until one final answer remains.

from llama_index.core.response_synthesizers import TreeSummarize

query_engine = index.as_query_engine(
    response_mode="tree_summarize",
    # or explicitly:
    # response_synthesizer=TreeSummarize(llm=llm)
)

If you have 20 nodes and a 4k context window that fits 4 nodes per call, tree_summarize makes 5 parallel calls (4 leaf summaries + 1 root merge) instead of 20 sequential ones. The trade-off: intermediate summaries lose detail the LLM might have used later.

Latency and throughput

With 10 nodes and a 2-second LLM call: refine takes ~20 seconds. Tree_summarize with 4-node batches takes ~6 seconds (2 leaf rounds + 1 root). The gap widens as node count grows.

If your SLA is sub-5-seconds and you retrieve 15+ nodes, refine is often a non-starter unless you aggressively truncate retrieval.

Token usage and cost

Refine sends the growing answer + next node each turn. Early iterations are cheap; later ones carry the full accumulated context. Total tokens ≈ Σ(context_per_node + answer_so_far). For long answers, this compounds.

Tree_summarize sends fixed-size batches. Each leaf call processes ~context_window tokens. Internal nodes process summaries, which are shorter. Total tokens ≈ (N/batch_size) × context_window + log₂(N) × summary_tokens.

In practice, tree_summarize uses 20-40% fewer tokens for the same node set because it never re-sends the full accumulated answer. But if your nodes are tiny (e.g., 200-token chunks), refine’s overhead is negligible and tree_summarize’s batch padding wastes tokens.

Answer quality: nuance vs. synthesis

Where refine wins

• Contradiction resolution: Later nodes can explicitly correct earlier ones. The LLM sees the full reasoning trail.
• Multi-hop reasoning: Questions requiring chaining facts across documents (“Company A acquired Company B, which had patent X — what’s the patent status?”) benefit from the iterative scratchpad.
• Long-form synthesis: Writing a report or comparison where structure matters. The LLM can outline, then fill sections iteratively.

Where tree_summarize wins

• Broad aggregation: “Summarize all customer feedback on pricing.” No single node contradicts another; you just need coverage.
• Fact extraction with redundancy: Many nodes say the same thing. Tree_summarize deduplicates naturally at the leaf level.
• High node counts: When you retrieve 50+ nodes, refine’s later iterations drown in context. Tree_summarize’s hierarchy preserves signal.

Where both struggle

Neither handles “needle in a haystack” well if the needle is in node 47 of 50. Refine might forget it by iteration 47. Tree_summarize might lose it in a leaf summary that deemed it irrelevant. For precise retrieval, consider compact mode (stuff everything into one prompt) or a reranker before synthesis.

Ergonomics and configuration

Both modes share the same ResponseSynthesizer interface. Key knobs:

from llama_index.core.response_synthesizers import Refine, TreeSummarize

# Refine knobs
refine = Refine(
    llm=llm,
    verbose=True,                    # logs each iteration
    response_builder=None,           # custom answer accumulator
)

# Tree summarize knaps
tree = TreeSummarize(
    llm=llm,
    summary_template=custom_tmpl,    # prompt for leaf summaries
    combine_template=custom_combine, # prompt for merging summaries
    verbose=True,
)

TreeSummarize exposes summary_template and combine_template separately — useful when leaf summaries need “extract key facts” but the root needs “write a cohesive answer.” Refine uses a single refine_template that receives {existing_answer} and {context_msg}.

Both respect service_context callbacks, so you get token counting, streaming handlers, and observability hooks identically.

Ecosystem and limits

• Streaming: Refine streams naturally — yield each iteration. Tree_summarize can only stream the final root call. If your UX needs progressive disclosure, refine wins.
• Custom prompts: Both support templating. Tree_summarize’s two-template design is more flexible for complex pipelines.
• Node limit: Refine hits context window hard limits around 15-20 nodes (depending on chunk size). Tree_summarize scales to hundreds — the tree depth grows logarithmically.
• Async: Both have async variants (arefine, atree_summarize). Tree_summarize’s leaf-level parallelism maps cleanly to asyncio.gather; refine’s sequential nature doesn’t benefit from async beyond single-call concurrency.
• Structured output: If you need JSON or Pydantic models, refine’s iterative approach lets you validate and repair at each step. Tree_summarize validates once at the end — simpler but all-or-nothing.

Comparison table

Which to choose: verdict by use case

Choose refine when:

• Multi-hop QA: The answer requires chaining facts across documents where later context must explicitly modify earlier conclusions.
• Long-form generation: Writing reports, comparisons, or analyses where structure and narrative flow matter.
• Low node count (< 10): Latency penalty is acceptable; quality gain is real.
• Streaming UX required: You need to show progressive reasoning to the user.
• Contradiction-heavy corpus: Legal, medical, or technical docs where later evidence overturns earlier.

Choose tree_summarize when:

• High-volume retrieval: 20+ nodes routinely. Refine will timeout or blow context.
• Broad summarization: “Summarize all Q3 earnings calls” — coverage > nuance.
• Latency SLA < 5s: Parallel leaf calls are your only path at scale.
• Token budget tight: 20-40% savings compound across thousands of queries.
• Redundant corpus: Many nodes overlap; tree deduplication helps.

Choose neither when:

• Precise fact lookup: Use compact mode with a reranker (top-3-5 nodes stuffed into one prompt). Both tree and refine add synthesis noise.
• Structured extraction: Use compact + Pydantic output parser. Iterative or hierarchical synthesis fights schema adherence.
• Real-time chat: Consider simple_summarize (truncate to fit) or a dedicated extractive QA model. Synthesis modes add 2-10x latency over single-pass.

Practical hybrid pattern

A common production pattern: retrieve 50 nodes, rerank to top 15, then refine. Or retrieve 100, cluster by topic, tree_summarize each cluster, then refine across cluster summaries.

# Two-stage: tree summarize clusters, then refine across clusters
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.response_synthesizers import TreeSummarize, Refine

# Stage 1: cluster nodes by embedding similarity (pseudo-code)
clusters = cluster_nodes(retrieved_nodes, k=5)

# Stage 2: tree summarize each cluster in parallel
cluster_summaries = await asyncio.gather(*[
    TreeSummarize(llm=llm).aget_response(query, cluster_nodes)
    for cluster_nodes in clusters
])

# Stage 3: refine across cluster summaries (now only 5 nodes)
final = Refine(llm=llm).get_response(query, cluster_summaries)

This gives you tree_summarize’s breadth at the leaf level and refine’s nuance at the integration level — often the best of both worlds.

Start with tree_summarize for any retrieval set above 15 nodes or any latency-sensitive path. Switch to refine only when you have evidence that iterative correction materially improves your eval metrics. Measure both on your actual queries — the theoretical trade-offs only matter if they show up in your specific corpus and question distribution.