LlamaIndex query pipelines vs query engines

If you’ve built anything non-trivial with LlamaIndex, you’ve hit the llamaindex query pipeline vs query engine comparison decision point. Query engines are the legacy abstraction — simple, opinionated, and sufficient for basic retrieval-augmented generation. Query pipelines are the newer DAG-based framework that lets you compose retrieval, reranking, synthesis, and post-processing as explicit, inspectable steps. This article breaks down where each shines, where they frustrate, and how to choose.

Core abstraction differences

Query engines follow the classic LlamaIndex pattern: you instantiate a retriever, maybe a response synthesizer, and the engine handles the loop. VectorStoreIndex.as_query_engine() returns an object with a .query() method. Internally it wires retriever → synthesizer → response. You configure behavior through constructor arguments — similarity_top_k, response_mode, node_postprocessors — and the engine hides the control flow.

Query pipelines make the control flow explicit. You define a QueryPipeline by chaining components: InputComponent → RetrieverComponent → RerankComponent → SynthesizerComponent → OutputComponent. Each component declares its inputs and outputs. The pipeline validates the DAG at construction time and executes it with a single run() call. You can insert branching, conditional logic, parallel retrieval, or custom components without subclassing.

# Query engine: opaque but concise
query_engine = index.as_query_engine(
    similarity_top_k=10,
    node_postprocessors=[SimilarityPostprocessor(similarity_cutoff=0.7)],
    response_mode="tree_summarize"
)
response = query_engine.query("What were the Q3 revenue drivers?")

# Query pipeline: explicit DAG
from llama_index.core.query_pipeline import QueryPipeline, InputComponent
from llama_index.core import PromptTemplate

pipeline = QueryPipeline(
    modules={
        "input": InputComponent(),
        "retriever": index.as_retriever(similarity_top_k=20),
        "reranker": CohereRerank(top_n=5),
        "synthesizer": get_response_synthesizer(
            response_mode="tree_summarize",
            summary_template=PromptTemplate(CUSTOM_TEMPLATE)
        ),
    },
    verbose=True
)
pipeline.add_chain(["input", "retriever", "reranker", "synthesizer"])
response = pipeline.run(query_str="What were the Q3 revenue drivers?")

The pipeline version is more verbose but exposes every stage. You can swap the reranker for a cross-encoder, add a hybrid retrieval branch, or insert a citation extractor without touching the synthesizer.

Capabilities and composability

Query engines cover the 80% case: single-retriever, single-synthesizer flows with optional post-processors. They support response_mode variants (refine, compact, tree_summarize, simple_summarize, accumulate) and a fixed set of node_postprocessors (similarity filter, keyword filter, rerankers). If your flow fits this mold, engines are faster to write.

Query pipelines handle the remaining 20% — and the 20% that grows into 50% as requirements accumulate. Common patterns engines struggle with:

• Hybrid retrieval: vector + BM25 in parallel, then reciprocal rank fusion
• Multi-hop reasoning: retrieve → generate sub-questions → retrieve again → synthesize
• Conditional routing: route to SQL index for structured queries, vector index for semantic ones
• Streaming with citations: emit tokens while attaching source nodes incrementally
• Custom component integration: plug in a PII scrubber, a fact-checker, or a cost tracker as a first-class node

# Hybrid retrieval in a pipeline — painful in a query engine
from llama_index.core.query_pipeline import QueryPipeline
from llama_index.retrievers.bm25 import BM25Retriever

pipeline = QueryPipeline(modules={
    "input": InputComponent(),
    "vector_retriever": vector_index.as_retriever(similarity_top_k=10),
    "bm25_retriever": BM25Retriever.from_defaults(index=bm25_index, similarity_top_k=10),
    "fusion": ReciprocalRerankFusion(top_n=5),
    "synthesizer": get_response_synthesizer(response_mode="compact"),
})
pipeline.add_chain(["input", "vector_retriever", "fusion"])
pipeline.add_chain(["input", "bm25_retriever", "fusion"])
pipeline.add_link("fusion", "synthesizer", src_key="nodes", dest_key="nodes")

Engines can do hybrid retrieval via QueryFusionRetriever, but you lose the ability to inspect intermediate results or inject logic between fusion and synthesis.

Ergonomics and debugging

Query engines win on initial velocity. One line gets you a working RAG endpoint. The API surface is small: query(), aquery(), streaming_response. Errors surface as exceptions from the synthesizer or retriever — familiar stack traces.

Query pipelines demand more upfront investment. You must understand the component protocol: each module declares input_keys and output_keys, and the pipeline validates connections. Miswired DAGs raise ValidationError at construction, not runtime. This catches bugs early but feels heavier during prototyping.

Debugging pipelines is better once you’re past the learning curve. verbose=True prints the execution graph with timing per node. You can hook callback_manager to log inputs/outputs at each stage. The QueryPipeline object is serializable — you can json.dumps(pipeline.to_dict()) and reconstruct it, which enables version-controlled pipeline definitions and A/B testing.

# Inspecting pipeline execution
pipeline = QueryPipeline(modules={...}, verbose=True)
response = pipeline.run(query_str="...")

# Output:
# [QueryPipeline] Starting run...
# [InputComponent] input: query_str="..."
# [VectorRetriever] retrieved 10 nodes in 42ms
# [CohereRerank] reranked to 5 nodes in 180ms
# [ResponseSynthesizer] synthesized in 1.2s
# [QueryPipeline] Run complete in 1.4s

Engines offer callback_manager too, but the internal steps are opaque — you see “retrieving” then “synthesizing” with no visibility into post-processor timing.

Latency and throughput

Both abstractions execute the same underlying components, so raw model latency is identical. The difference is overhead and parallelization.

Query engines run sequentially: retrieve → post-process → synthesize. No parallelism within a single query. If you use QueryFusionRetriever, it parallelizes sub-retrievers internally, but you can’t parallelize retrieval with, say, a classification step.

Query pipelines can express parallel branches explicitly. The executor runs independent branches concurrently (using asyncio.gather under the hood for async components). For a hybrid retrieval + classification flow:

pipeline = QueryPipeline(modules={
    "input": InputComponent(),
    "classifier": QueryClassifier(),           # runs first
    "vector_retriever": vector_retriever,      # runs in parallel with bm25
    "bm25_retriever": bm25_retriever,
    "router": RouteSelector(),                 # picks branch based on classifier
    "synthesizer": synthesizer,
})
# Classifier runs, then vector+bm25 run in parallel, then router picks, then synthesize

In practice, pipeline parallelism saves 200-500ms on complex flows where you’d otherwise chain sequentially. For single-retriever flows, the overhead is negligible (~5-10ms).

Throughput under load: both scale horizontally the same way — stateless workers, async endpoints. Pipelines have a slight edge because you can swap components (e.g., a faster retriever for high-QPS tiers) without changing the pipeline structure.

Ecosystem and integrations

Query engines have deeper integration with LlamaIndex’s higher-level abstractions: ChatEngine, AgentRunner, SubQuestionQueryEngine. These expect a BaseQueryEngine interface. If you’re building a chat agent that delegates to tools, you’ll likely wrap a query engine, not a pipeline.

Query pipelines are newer (stable since v0.10) and the ecosystem is catching up. QueryPipeline implements BaseQueryEngine via as_query_engine(), so you can plug a pipeline into an agent — but the pipeline’s streaming support is still maturing. As of writing, streaming_response on a pipeline-backed engine works for token streaming but not for incremental citation emission.

Third-party integrations (LangChain, Haystack, custom wrappers) typically target the query engine interface. If you’re interoperating with non-LlamaIndex tooling, engines are safer.

Limits and gotchas

Gotchas specific to engines:

• response_mode="tree_summarize" can blow context window on large top_k — no automatic truncation
• node_postprocessors run after retrieval but before synthesis; you can’t post-process synthesized chunks
• No built-in way to short-circuit (e.g., “if confidence < 0.3, return fallback”)

Gotchas specific to pipelines:

• Cyclic dependencies raise at construction, but dynamic cycles (loop until condition) require QueryPipeline + custom loop component — not ergonomic yet
• InputComponent only accepts keyword args matching declared input_keys; positional args fail silently in some versions
• Component output_keys must match downstream input_keys exactly — typo = validation error

Which to choose

Start with a query engine if:

• You need a working RAG endpoint today and the flow is retrieve → rerank → synthesize
• You’re building a chat agent or sub-question engine that expects BaseQueryEngine
• Your team is new to LlamaIndex and you want minimal conceptual overhead
• You rely on streaming citations in production

Move to a query pipeline when:

• You need hybrid retrieval, multi-hop, or conditional routing
• You want to version-control and A/B test retrieval strategies as JSON
• You need per-stage observability (latency, token counts, node counts) without custom callbacks
• You’re building a platform where non-engineers configure retrieval via UI → pipeline JSON
• You anticipate swapping components (rerankers, synthesizers) frequently

Migration path: You don’t have to choose once. Wrap a pipeline as a query engine for agent compatibility:

# Pipeline as drop-in query engine for agents
pipeline = QueryPipeline(modules={...})
query_engine = pipeline.as_query_engine()

# Now usable in SubQuestionQueryEngine, AgentRunner, etc.
sub_question_engine = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=[QueryEngineTool(query_engine=query_engine, ...)]
)

This lets you migrate incrementally: keep engines for simple tools, promote complex ones to pipelines, and compose both in the same agent.

The llamaindex query pipeline vs query engine comparison ultimately comes down to how much control flow you need to express. Engines are a sensible default; pipelines are the tool you reach for when the default constrains you. Most production systems end up with both — engines for the simple paths, pipelines for the complex ones — and that’s the right architecture.