LlamaIndex response modes: compact, refine, tree summarize

If you’re building a RAG system with LlamaIndex, the response mode you choose for your query engine directly controls the trade-off between answer quality, latency, and token spend. The three built-in modes — compact, refine, and tree_summarize — each implement a fundamentally different strategy for synthesizing retrieved nodes into a final answer. This post breaks down how they work, when each one wins, and how to pick without guessing.

How each response mode works

Compact mode

Compact mode is the default for a reason: it stuffs as many retrieved nodes as possible into a single LLM call, up to the model’s context window. LlamaIndex concatenates the node texts with a separator, prepends the query and a synthesis prompt, and sends one request. If the combined text exceeds the context limit, it truncates the node list (by default, keeping the highest-scored nodes first).

from llama_index.core import VectorStoreIndex
from llama_index.core.query_engine import RetrieverQueryEngine

index = VectorStoreIndex.from_documents(docs)
query_engine = index.as_query_engine(
    response_mode="compact",
    similarity_top_k=10,
)
response = query_engine.query("What were the Q3 revenue drivers?")

The synthesis prompt instructs the model to answer using only the provided context. You get one LLM call, one round-trip, and a single billable completion.

Refine mode

Refine mode takes an iterative approach. It sends the first node (plus the query) to the LLM to produce an initial answer. Then for each subsequent node, it sends the previous answer, the new node, and a refine prompt asking the model to update or extend the answer. This continues until all nodes are processed or a max-iterations limit is hit.

query_engine = index.as_query_engine(
    response_mode="refine",
    similarity_top_k=10,
)

The refine prompt typically looks like: “Given the existing answer and new context, refine the answer to better address the query.” This means N LLM calls for N nodes — linear latency and cost growth.

Tree summarize mode

Tree summarize builds a bottom-up reduction tree. It groups nodes into batches that fit within the context window, summarizes each batch in parallel (one LLM call per batch), then recursively summarizes the summaries until a single root answer remains. The branching factor is determined by how many nodes fit per call.

query_engine = index.as_query_engine(
    response_mode="tree_summarize",
    similarity_top_k=20,
)

This mode is the only one that can meaningfully process large node sets (20+) without truncation, because it never tries to stuff everything into one context window.

Comparison across dimensions

Capabilities and quality characteristics

Compact: precision over breadth

Compact mode shines when your retrieval is already tight — high-precision top-k, focused questions, and a context window that comfortably fits the nodes. The model sees all evidence at once, which helps it resolve contradictions and weigh conflicting sources in a single reasoning pass.

The failure mode is silent truncation. If you set similarity_top_k=20 on a 4k-context model with 512-token chunks, you’ll lose the bottom half of your retrieval without a warning. Always check response.source_nodes length versus your top_k.

# Debug: verify how many nodes actually made it into the prompt
print(f"Retrieved: {len(response.source_nodes)}")
# If this < similarity_top_k, you're truncating

Refine: narrative synthesis and iterative correction

Refine excels when the answer requires stitching together a narrative from chronologically or logically ordered nodes — think “trace the decision process” or “summarize the evolution of this policy.” Each iteration can correct hallucinations from earlier steps because the model sees its own previous output.

The cost is brutal: 10 nodes = 10 LLM calls. At 2-3 seconds per call, you’re looking at 20-30 second latency. Token spend scales linearly. Refine also tends to drift — early errors compound unless the refine prompt explicitly instructs correction.

from llama_index.core.prompts import PromptTemplate

refine_template = PromptTemplate(
    "Original query: {query_str}\n"
    "Existing answer: {existing_answer}\n"
    "New context: {context_msg}\n"
    "Refine the existing answer using the new context. "
    "If the new context contradicts the existing answer, "
    "explicitly note the contradiction and resolve it."
)
query_engine = index.as_query_engine(
    response_mode="refine",
    refine_template=refine_template,
)

Tree summarize: scale and coverage

Tree summarize is the only mode that handles large retrieval sets without truncation. It’s designed for “summarize this entire codebase” or “what are all the risks mentioned across 50 documents?” queries where you need broad coverage.

The trade-off is loss of fine-grained cross-node reasoning. Each batch summarizes independently, so contradictions across batches may not be resolved until the final reduction step — if they survive that far. The tree structure also means you can’t stream partial results.

# Control batch size via summary_template and chunk_size
from llama_index.core import Settings
Settings.chunk_size = 1024  # affects how many nodes fit per batch

Latency and cost modeling

For a concrete sense of the economics, assume a typical setup: 1k-token chunks, 4k output limit, gpt-4o-mini pricing ($0.15/M in, $0.60/M out).

These are rough estimates — actual latency depends on provider, region, and load. The key insight: refine scales linearly, tree summarize scales logarithmically, compact is constant but loses data.

If you’re routing through a gateway like n4n.ai that implements automatic fallback across providers, the latency variance across sequential refine calls becomes more pronounced — each retry adds tail latency. Tree summarize’s parallel batches absorb this better.

Ergonomics and customization

All three modes accept custom prompt templates, but the extension points differ:

• Compact: text_qa_template — single prompt with {context_str} and {query_str}
• Refine: text_qa_template (initial) + refine_template (iterative) with {existing_answer}, {context_msg}, {query_str}
• Tree summarize: summary_template (batch summarization) + text_qa_template (final reduction)

Refine gives you the most control over the reasoning process because you can inject instructions at each iteration. Tree summarize’s summary_template controls the batch-level compression — critical for preserving the right details.

# Tree summarize: customize batch summary to preserve entities
summary_template = PromptTemplate(
    "Extract key facts, figures, and entity relationships from the context "
    "that are relevant to: {query_str}\n"
    "Context: {context_str}\n"
    "Concise summary:"
)

Limits and gotchas

Compact mode truncation is silent

The most common production bug: increasing similarity_top_k to improve recall, but silently losing nodes to truncation. The fix is either reducing chunk size, increasing context window (if your model supports it), or switching modes.

# Guardrail: warn if truncation occurred
if len(response.source_nodes) < query_engine.similarity_top_k:
    logger.warning(f"Truncated: {len(response.source_nodes)}/{query_engine.similarity_top_k} nodes used")

Refine mode has no parallelism

You cannot parallelize refine — each step depends on the previous answer. If you need lower latency with refine-style quality, consider compact_accumulate (a variant that accumulates context across calls but still sequential) or restructure your retrieval to return fewer, higher-quality nodes.

Tree summarize loses node attribution

Because batches are summarized independently, the final answer’s source_nodes may not map cleanly to specific claims. If you need citation-grade attribution, compact or refine with explicit citation prompts work better.

# Compact with citation prompt
citation_template = PromptTemplate(
    "Answer using only the context. Cite sources like [1], [2].\n"
    "Context:\n{context_str}\n"
    "Query: {query_str}\n"
    "Answer:"
)

Streaming constraints

Compact supports streaming out of the box. Refine can stream each iteration but the final answer only arrives after the last step. Tree summarize cannot stream — the reduction tree must complete before any output.

# Streaming with compact
streaming_engine = index.as_query_engine(
    response_mode="compact",
    streaming=True,
)
for token in streaming_engine.query("...").response_gen:
    print(token, end="", flush=True)

Which to choose: verdict by use case

Choose compact when:

• Your retrieval is high-precision (top-5 to top-10 covers the answer)
• Latency and cost are primary constraints
• You need streaming responses
• The query is fact-seeking or single-hop (“What was Q3 revenue?”)
• You’re using a model with a large context window (128k+) and moderate chunk sizes

Choose refine when:

• The answer requires narrative synthesis across ordered nodes
• You need iterative correction (e.g., legal review, policy compliance)
• Node count is small (≤10) and you can absorb the latency/cost
• You want explicit control over the reasoning loop via prompts
• Streaming partial progress is valuable to your UX

Choose tree_summarize when:

• You need broad coverage over many nodes (20+)
• The query is “summarize all X” or “what are all Y mentioned”
• Retrieval recall matters more than precision
• You can tolerate 2-3x compact latency for completeness
• You don’t need streaming or fine-grained citations

Hybrid strategy for production

Most production systems don’t pick one mode globally. They route by query type:

from llama_index.core.query_engine import RouterQueryEngine
from llama_index.core.selectors import LLMSingleSelector
from llama_index.core.tools import QueryEngineTool

compact_engine = index.as_query_engine(response_mode="compact", similarity_top_k=8)
refine_engine = index.as_query_engine(response_mode="refine", similarity_top_k=6)
tree_engine = index.as_query_engine(response_mode="tree_summarize", similarity_top_k=30)

router = RouterQueryEngine(
    selector=LLMSingleSelector.from_defaults(),
    query_engine_tools=[
        QueryEngineTool.from_defaults(
            query_engine=compact_engine,
            description="Fact-seeking questions, specific lookups, single-hop queries"
        ),
        QueryEngineTool.from_defaults(
            query_engine=refine_engine,
            description="Narrative synthesis, chronological tracing, iterative reasoning needed"
        ),
        QueryEngineTool.from_defaults(
            query_engine=tree_engine,
            description="Broad summaries, 'list all', 'summarize entire', high-recall queries"
        ),
    ]
)

This lets the LLM classify the incoming query and dispatch to the right mode — compact for 80% of traffic, refine for the 15% that need narrative, tree_summarize for the 5% that need breadth.

Final note

The response mode is a lever, not a religion. Start with compact, measure truncation rate and answer quality on your eval set, then add refine or tree_summarize for the query classes that fail. Instrument len(response.source_nodes) vs similarity_top_k in production — that single metric tells you whether compact is silently dropping evidence.