n4nAI
Open console →
Framework tutorialsGuideLlamaIndex Query Engines for RAG

Sub-question query engine in LlamaIndex explained

A practical guide to LlamaIndex's SubQuestionQueryEngine — when to use it, how it decomposes complex queries, and production patterns for multi-document RAG.

n4n Team4 min read805 words

Audio narration

Coming soon — every post will get a voice note here.

The SubQuestionQueryEngine is LlamaIndex’s answer to multi-hop reasoning across heterogeneous data sources. Instead of stuffing everything into a single retrieval call, it decomposes a complex query into targeted sub-questions, routes each to the appropriate index, then synthesizes a final answer. This guide walks through the mechanics, a working implementation, and the failure modes you’ll hit in production.

When you actually need sub-question decomposition

Single-index retrieval works fine when your corpus is homogeneous — one set of docs, one embedding space, one retrieval strategy. Things break when:

  • Different document types need different retrieval: PDFs with tables vs. API specs vs. Slack threads
  • Queries span distinct knowledge domains: “Compare the 2023 revenue of Acme Corp with their main competitor’s Q4 earnings” requires two separate lookups
  • You need comparative or aggregative reasoning: “Which of these three vendors supports SOC2 Type II and has Python SDKs?” isn’t a single vector search

The SubQuestionQueryEngine handles this by generating a query plan, executing each sub-question against its designated tool, and combining results. It’s not magic — it’s an LLM-driven planner plus a fixed execution loop.

Core architecture

User Query
    │
    ▼
┌─────────────────────┐
│  Question Generator │  (LLM: "break this into sub-questions")
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   Query Planner     │  (maps sub-questions → QueryEngineTools)
└──────────┬──────────┘
           │
           ▼
    ┌──────┴──────┐
    ▼             ▼
┌───────┐    ┌───────┐
│ Tool A│    │ Tool B│  (each is a QueryEngineTool wrapping an index)
└───┬───┘    └───┬───┘
    │            │
    └─────┬──────┘
          ▼
┌─────────────────────┐
│  Response Synthesizer│ (LLM: "combine these answers")
└─────────────────────┘

Each QueryEngineTool wraps a query engine with a name and description. The planner uses those descriptions to route sub-questions. This is the key abstraction: tools are self-describing retrieval endpoints.

Minimal working example

from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    Settings,
)
from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.core.query_engine import SubQuestionQueryEngine
from llama_index.llms.openai import OpenAI

# Two distinct corpora
financial_docs = SimpleDirectoryReader("./data/financial").load_data()
technical_docs = SimpleDirectoryReader("./data/technical").load_data()

financial_index = VectorStoreIndex.from_documents(financial_docs)
technical_index = VectorStoreIndex.from_documents(technical_docs)

# Wrap each index as a tool with a clear description
financial_tool = QueryEngineTool(
    query_engine=financial_index.as_query_engine(similarity_top_k=3),
    metadata=ToolMetadata(
        name="financial_reports",
        description="Use for revenue, earnings, expenses, and financial metrics questions",
    ),
)

technical_tool = QueryEngineTool(
    query_engine=technical_index.as_query_engine(similarity_top_k=3),
    metadata=ToolMetadata(
        name="technical_specs",
        description="Use for API specifications, architecture diagrams, and implementation details",
    ),
)

# Build the sub-question engine
llm = OpenAI(model="gpt-4o-mini", temperature=0)
engine = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=[financial_tool, technical_tool],
    llm=llm,
    verbose=True,
)

response = engine.query(
    "What was Acme's Q3 2023 revenue and which API version introduced the new auth flow?"
)
print(response)

The verbose=True flag prints the generated sub-questions and tool routing — essential for debugging.

How the question generator works

By default, SubQuestionQueryEngine uses a QuestionGenerator prompt that looks like this:

from llama_index.core.question_gen import LLMQuestionGenerator
from llama_index.core.prompts import PromptTemplate

CUSTOM_QUESTION_GEN_PROMPT = PromptTemplate(
    "Given the user question: {query_str}\n"
    "And the available tools:\n{tool_descriptions}\n"
    "Generate 2-4 specific sub-questions that can be answered by these tools.\n"
    "Each sub-question must target exactly one tool.\n"
    "Return as a JSON list of objects: {\"sub_question\": \"...\", \"tool_name\": \"...\"}"
)

question_generator = LLMQuestionGenerator.from_defaults(
    llm=llm,
    prompt_template=CUSTOM_QUESTION_GEN_PROMPT,
)

engine = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=[financial_tool, technical_tool],
    llm=llm,
    question_generator=question_generator,
)

The default prompt works for simple cases. Customize it when:

  • You need to constrain the number of sub-questions
  • Tool descriptions are ambiguous and the LLM routes incorrectly
  • You want to enforce a specific decomposition strategy (e.g., always separate “what” from “why”)

Response synthesis: the hidden bottleneck

The final synthesis step receives all sub-question answers and the original query. The default prompt essentially says “here are the answers, write a response.” This fails when:

  • Sub-answers contradict each other
  • One sub-question returns “I don’t know” but others have data
  • The user asked for a comparison but the synthesizer just concatenates

Override the response synthesizer:

from llama_index.core.response_synthesizers import CompactAndRefine
from llama_index.core.prompts import PromptTemplate

SYNTHESIS_PROMPT = PromptTemplate(
    "Original question: {query_str}\n\n"
    "Sub-question answers:\n{context_str}\n\n"
    "Instructions:\n"
    "1. If any sub-question could not be answered, state that explicitly.\n"
    "2. For comparative questions, structure as a table or clear comparison.\n"
    "3. Cite which tool provided each fact.\n"
    "4. Do not hallucinate connections between unrelated answers.\n\n"
    "Final answer:"
)

synthesizer = CompactAndRefine(
    llm=llm,
    text_qa_template=SYNTHESIS_PROMPT,
    verbose=True,
)

engine = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=[financial_tool, technical_tool],
    llm=llm,
    response_synthesizer=synthesizer,
)

Common pitfalls and how to fix them

1. Tool description ambiguity

Symptom: Sub-questions route to the wrong tool, or the planner generates duplicate sub-questions for the same tool.

Fix: Make descriptions mutually exclusive and specific. Bad: “Use for financial data.” Good: “Use for revenue, profit, expenses, and balance sheet figures from SEC filings and earnings reports.” Include negative constraints: “Do not use for technical architecture questions.”

2. Sub-question explosion

Symptom: A simple query generates 8+ sub-questions, blowing up latency and cost.

Fix: Constrain the question generator. Add "Generate at most 3 sub-questions" to the prompt. Or implement a custom QuestionGenerator that uses a classifier first:

from llama_index.core.question_gen.base import BaseQuestionGenerator
from typing import List
from llama_index.core.tools import ToolMetadata
from llama_index.core.schema import QueryBundle

class ClassifierQuestionGenerator(BaseQuestionGenerator):
    def __init__(self, llm, tools: List[ToolMetadata], max_subquestions: int = 3):
        self.llm = llm
        self.tools = tools
        self.max_subquestions = max_subquestions

    def generate(self, query_bundle: QueryBundle, **kwargs) -> List[QueryBundle]:
        # First classify which tools are relevant
        tool_names = [t.name for t in self.tools]
        classification_prompt = f"""
        Query: {query_bundle.query_str}
        Available tools: {tool_names}
        Return ONLY the tool names that are relevant, comma-separated.
        """
        relevant = self.llm.complete(classification_prompt).text.strip().split(", ")
        relevant = [t for t in relevant if t in tool_names][:self.max_subquestions]

        # Then generate one sub-question per relevant tool
        sub_questions = []
        for tool_name in relevant:
            tool = next(t for t in self.tools if t.name == tool_name)
            sq_prompt = f"""
            Query: {query_bundle.query_str}
            Tool: {tool_name} - {tool.description}
            Generate ONE specific sub-question for this tool.
            """
            sub_q = self.llm.complete(sq_prompt).text.strip()
            sub_questions.append(QueryBundle(query_str=sub_q))
        return sub_questions

3. Lost context in synthesis

Symptom: The final answer misses nuance because the synthesizer only sees condensed sub-answers.

Fix: Use TreeSummarize instead of CompactAndRefine for the final synthesis, or pass raw retrieval nodes through via a custom response synthesizer that preserves citations.

from llama_index.core.response_synthesizers import TreeSummarize

synthesizer = TreeSummarize(
    llm=llm,
    summary_template=SYNTHESIS_PROMPT,
    verbose=True,
)

4. No fallback for failed sub-questions

Symptom: One tool errors or returns empty results, and the whole pipeline produces a generic “I couldn’t find information” response.

Fix: Wrap each tool’s query engine with error handling:

from llama_index.core.query_engine import BaseQueryEngine
from llama_index.core.schema import NodeWithScore
from typing import List

class ResilientQueryEngine(BaseQueryEngine):
    def __init__(self, base_engine, tool_name: str):
        self.base_engine = base_engine
        self.tool_name = tool_name

    def query(self, query_bundle):
        try:
            return self.base_engine.query(query_bundle)
        except Exception as e:
            # Return a structured "failed" response the synthesizer can handle
            from llama_index.core.response.schema import Response
            return Response(
                response=f"[{self.tool_name} ERROR: {str(e)}]",
                source_nodes=[],
                metadata={"error": True, "tool": self.tool_name},
            )

    async def aquery(self, query_bundle):
        return self.query(query_bundle)

# Wrap tools
financial_tool = QueryEngineTool(
    query_engine=ResilientQueryEngine(
        financial_index.as_query_engine(), "financial_reports"
    ),
    metadata=ToolMetadata(...),
)

Then update your synthesis prompt to handle [TOOL ERROR: ...] markers explicitly.

Advanced: Dynamic tool selection

Static tool lists work when your corpus is fixed. For dynamic environments — multi-tenant RAG, per-user document collections — you need to construct tools at query time.

from llama_index.core import VectorStoreIndex
from llama_index.core.tools import QueryEngineTool, ToolMetadata

def build_subquestion_engine(user_id: str, query: str):
    # Fetch user's available indexes from your registry
    user_indexes = get_user_indexes(user_id)  # your implementation

    tools = []
    for idx_info in user_indexes:
        index = load_index(idx_info.index_id)
        tools.append(
            QueryEngineTool(
                query_engine=index.as_query_engine(similarity_top_k=4),
                metadata=ToolMetadata(
                    name=idx_info.tool_name,
                    description=idx_info.description,
                ),
            )
        )

    return SubQuestionQueryEngine.from_defaults(
        query_engine_tools=tools,
        llm=OpenAI(model="gpt-4o-mini"),
    )

This pattern lets you scale to thousands of isolated corpora without a monolithic index.

Evaluating sub-question quality

You can’t improve what you don’t measure. Log these for every query:

import json
from datetime import datetime

def log_subquestion_execution(engine, query_str: str, response):
    log_entry = {
        "timestamp": datetime.utcnow().isoformat(),
        "query": query_str,
        "sub_questions": [
            {
                "sub_question": sq.query_str,
                "tool_used": getattr(sq, "tool_name", "unknown"),
                "answer": str(sq.response) if hasattr(sq, "response") else "N/A",
                "source_count": len(getattr(sq, "source_nodes", [])),
            }
            for sq in getattr(response, "sub_questions", [])
        ],
        "final_answer_length": len(str(response)),
        "total_latency_ms": getattr(response, "metadata", {}).get("total_latency_ms"),
    }
    print(json.dumps(log_entry))  # Ship to your observability stack

Track:

  • Sub-questions per query (distribution)
  • Tool routing accuracy (manual spot-check)
  • Failure rate per tool
  • Synthesis quality (human eval on a sample)

When not to use SubQuestionQueryEngine

  • Single homogeneous corpus: Standard VectorStoreIndex.as_query_engine() is faster and cheaper
  • Simple lookups: “What is the capital of France?” doesn’t need decomposition
  • Latency-critical paths: The planner + multiple retrievals + synthesis adds 2-5x latency vs. single retrieval
  • No clear tool boundaries: If you can’t write crisp, non-overlapping tool descriptions, the planner will hallucinate routes

Production checklist

Before shipping:

  • Tool descriptions are mutually exclusive and tested against a query set
  • Question generator constrained to max 3-4 sub-questions
  • Response synthesizer handles missing/contradictory sub-answers
  • Each tool wrapped with error handling and timeout
  • Logging captures sub-question trace for debugging
  • Latency budget allocated (plan for 3-8s p95 with gpt-4o-mini)
  • Fallback to single-index retrieval when only one tool is relevant

The SubQuestionQueryEngine is a solid primitive for multi-source RAG. Treat it like any distributed system: explicit contracts (tool descriptions), observability at every hop, and graceful degradation when a shard fails.

Tagsllamaindexquery-engineragsub-question

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Framework tutorialsHybrid search query engine in LlamaIndex tutorialBuild a production-ready hybrid search query engine in LlamaIndex combining vector similarity and BM25 keyword search with runnable code and tuning guidance.Framework tutorialsBuild a citation-aware query engine in LlamaIndexHands-on llamaindex citation query engine tutorial: build a RAG pipeline with LlamaIndex that returns answers annotated with verifiable source citations.Framework tutorialsBuild a RAG query engine in LlamaIndex with n4n.aiStep-by-step llamaindex rag query engine tutorial: install LlamaIndex, load docs, configure LLM via n4n.ai, build vector index, and run queries.Framework tutorialsLlamaIndex VectorStoreIndex query engine explainedA practical llamaindex vectorstoreindex query engine tutorial: build the index, configure models, tune retrieval, run RAG queries, and avoid common pitfalls.Framework tutorialsLlamaIndex query pipelines vs query enginesCompare LlamaIndex query pipelines and query engines across capabilities, ergonomics, latency, and ecosystem to pick the right abstraction for your RAG system.Framework tutorialsLlamaIndex tree summarize vs refine response modeCompare LlamaIndex tree_summarize vs refine response modes for RAG — latency, cost, quality trade-offs, and when to use each.

All llamaindex query engines for rag posts →