LlamaIndex query engines vs agents: when to use each

The choice between a llamaindex query engine vs agent determines how many LLM round-trips you pay for, how much control you retain, and whether your system can act on external state. Both build on the same LlamaIndex retrieval primitives, but they diverge sharply in execution model: one is a deterministic-ish pipeline, the other is an LLM-driven control loop.

Capabilities

A query engine answers a question by retrieving relevant nodes from an index and synthesizing a response. The simplest form is a single vector search plus one generation call. LlamaIndex extends this with query transforms (e.g., HybridRetriever, SubQuestionQueryEngine) that decompose a complex question into sub-queries, but the flow remains bounded and retrieval-centric.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

docs = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(docs)
query_engine = index.as_query_engine(similarity_top_k=3)
response = query_engine.query("What is the refund policy?")
print(response)

An agent wraps the LLM in a reasoning loop. It decides which tools to call, observes outputs, and iterates until it satisfies the user intent. Tools can be query engines, web search, Python interpreters, or internal APIs. The agent can plan across multiple turns and maintain conversational state.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.agents import ReActAgent
from llama_index.core.tools import QueryEngineTool, ToolMetadata

docs = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(docs)
qe = index.as_query_engine()
qe_tool = QueryEngineTool(
    query_engine=qe,
    metadata=ToolMetadata(name="policy_search", description="Search refund policy")
)

agent = ReActAgent.from_tools([qe_tool], llm=llm, verbose=True)
agent.chat("Compare our refund policy to competitor X using web search")

The llamaindex query engine vs agent distinction is fundamentally about whether the LLM controls the execution graph or merely fills a leaf node in yours.

Price/cost model

Query engines have predictable token spend. A standard RAG call consumes tokens for the retrieved context plus the generated answer. Even with sub-question decomposition, you typically cap at a small multiple of that baseline.

Agents are open-ended. Each reasoning step emits a completion, then ingests tool results back into the prompt. A task that takes eight steps can cost eight times the context tokens of a single query-engine call, plus the tokens from tool payloads. Without a max_iterations guard, a confused agent will burn tokens in circles.

If you route both patterns through an OpenAI-compatible gateway such as n4n.ai, you get per-token metering and automatic fallback across 240+ models, which flattens the cost variance when an agent spikes token usage mid-session.

Latency/throughput

A query engine’s p95 latency is the sum of vector lookup and one generation. For a 2k-token context and a 300-token answer on a mid-size model, that is often sub-second to a few seconds.

An agent’s latency is cumulative and serial. Step N cannot start until step N-1’s tool returns and the LLM processes it. A five-step task will feel at least five times slower than a query engine, even if individual LLM calls are fast. Throughput per GPU/endpoint drops because each agent request holds a session open longer.

Ergonomics

Query engines are declarative. You set similarity_top_k, response_mode, maybe a node_postprocessor, and you’re done. Unit tests are straightforward: feed a question, assert on output or cited nodes.

Agents require you to define tool schemas, handle tool errors, and decide termination. Debugging means inspecting a trace of thoughts and actions. LlamaIndex’s verbose=True helps, but you’ll still need logging to understand why an agent called the same tool four times.

# Agent tool contract matters
def get_order_status(order_id: str) -> str:
    """Fetch status for a given order id."""
    return external_api.get(order_id)

A weakly described tool leads to agents that misuse it or hallucinate arguments.

Ecosystem

Both consume the same LlamaIndex data loaders, embedding models, and vector stores (Chroma, Pinecone, Weaviate, etc.). Query engines drop directly into a ChatEngine or a simple REST endpoint.

Agents compose with the broader tool ecosystem. You can wrap a query engine as a QueryEngineTool and hand it to an agent alongside a SQL tool and a Python tool. This makes the agent the orchestration layer, while the query engine remains a specialist retriever.

Limits

Query engines fail on tasks requiring external state changes or multi-hop reasoning that exceeds their decomposition strategy. They cannot book a flight or call a webhook; they can only describe how based on retrieved text.

Agents suffer from non-determinism, loop traps, and prompt bloat. A retrieved document that confuses the model can derail the whole session. They also complicate compliance: every tool call is a potential side effect that must be audited.

Head-to-head summary

Which to choose

Use a LlamaIndex query engine when:

• You need answer grounded in internal documents with citation.
• Traffic is high-volume and cost per request must stay flat.
• The task is a single question or a decomposable query with no side effects.
• You want deterministic behavior for regression testing.

Use an agent when:

• The task requires calling multiple systems (search, database, API) in sequence.
• The user asks “find the best plan for me and sign me up” — action needed.
• You are building a conversational assistant that must adapt to missing info.
• The reasoning path is not known at design time and must be discovered.

Hybrid pattern: Ship a query engine as the default path, and escalate to an agent only when a confidence score or a parse failure indicates the question is too complex. This keeps 90% of requests cheap while preserving capability for the long tail.

In practice, the llamaindex query engine vs agent decision is not either/or across a product, but per-route. Start with a query engine, measure where it fails, and introduce agents only at those edges.