LlamaIndex QueryEngineTool: turning RAG into a tool

If you’ve built a RAG pipeline with LlamaIndex, you already have a query engine. The next step — making that engine callable by an agent — is where most tutorials stop. This llamaindex queryenginetool tutorial walks through the practical patterns: wrapping single and multiple indexes, handling metadata filters at tool-call time, and keeping latency predictable when an agent decides which tool to invoke.

What QueryEngineTool actually does

QueryEngineTool is a thin adapter. It takes any BaseQueryEngine — vector, keyword, SQL, or a custom hybrid — and exposes it through the FunctionTool interface that LlamaIndex agents expect. The tool name, description, and parameter schema come from you; the engine does the retrieval and synthesis.

from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.core.query_engine import VectorStoreIndex

index = VectorStoreIndex.from_documents(docs)
engine = index.as_query_engine(similarity_top_k=4)

tool = QueryEngineTool(
    query_engine=engine,
    metadata=ToolMetadata(
        name="policy_docs",
        description="Search the employee policy handbook for benefits, leave, and compliance questions.",
    ),
)

The agent sees a function called policy_docs that accepts a single string argument input (the query). Internally, the tool runs engine.query(input) and returns the response text. That’s it — no magic, just a consistent interface.

Single-index tool: the baseline pattern

Start with one well-tuned index. The tool description is your prompt engineering surface: it tells the LLM when to call this tool and how to phrase the query.

from llama_index.core.agent import ReActAgent
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o-mini", temperature=0)
agent = ReActAgent.from_tools([tool], llm=llm, verbose=True)

response = agent.chat("What's our parental leave policy for new hires?")
print(response)

Pitfall: vague descriptions. “Search documents” teaches the agent nothing. “Search the employee policy handbook for benefits, leave, and compliance questions” gives the model a decision boundary. If you have multiple tools, the description is the router.

Tradeoff: a single tool is simple but brittle. If the index covers policies and engineering runbooks, the agent will retrieve irrelevant chunks for half the queries. Split indexes before you split tools.

Multi-tool routing: one agent, many indexes

Real systems need multiple tools — policies, runbooks, API specs, ticket history. Each gets its own index and tool. The agent chooses at runtime.

from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.core.tools import QueryEngineTool, ToolMetadata

# Separate indexes, separate persistence
policy_index = VectorStoreIndex.from_documents(policy_docs)
runbook_index = VectorStoreIndex.from_documents(runbook_docs)
spec_index = VectorStoreIndex.from_documents(api_spec_docs)

tools = [
    QueryEngineTool(
        query_engine=policy_index.as_query_engine(similarity_top_k=4),
        metadata=ToolMetadata(
            name="policy_docs",
            description="HR policies: benefits, leave, compliance, onboarding.",
        ),
    ),
    QueryEngineTool(
        query_engine=runbook_index.as_query_engine(similarity_top_k=6),
        metadata=ToolMetadata(
            name="runbooks",
            description="Operational runbooks: incident response, deploy checklists, rollback procedures.",
        ),
    ),
    QueryEngineTool(
        query_engine=spec_index.as_query_engine(similarity_top_k=4),
        metadata=ToolMetadata(
            name="api_specs",
            description="OpenAPI specs and SDK reference for internal services.",
        ),
    ),
]

agent = ReActAgent.from_tools(tools, llm=llm, verbose=True)

Common failure mode: overlapping descriptions. If two tools claim “technical documentation,” the agent picks randomly. Make descriptions mutually exclusive. Test by asking borderline questions (“How do I deploy the payments service?”) and verify the right tool fires.

Latency note: each tool call is a full retrieve + synthesize round trip. Three tools means up to three sequential LLM calls if the agent checks multiple. Keep similarity_top_k tight and consider response_mode="compact" on the query engine to reduce synthesis tokens.

Passing metadata filters at tool-call time

Static indexes work until you need tenant isolation, version pinning, or date ranges. The tool interface only accepts a string query — but you can embed filter instructions in the query and parse them in a custom query engine.

from llama_index.core.query_engine import BaseQueryEngine
from llama_index.core.vector_stores import MetadataFilters, ExactMatchFilter
from llama_index.core import QueryBundle
from typing import Optional

class FilteredQueryEngine(BaseQueryEngine):
    def __init__(self, base_engine, default_filters: Optional[MetadataFilters] = None):
        self.base_engine = base_engine
        self.default_filters = default_filters

    def _parse_filters(self, query_str: str) -> tuple[str, MetadataFilters]:
        # Convention: "tenant:acme corp | query text"
        if " | " in query_str:
            prefix, real_query = query_str.split(" | ", 1)
            filters = MetadataFilters(filters=[])
            for part in prefix.split(","):
                if ":" in part:
                    k, v = part.split(":", 1)
                    filters.filters.append(ExactMatchFilter(key=k.strip(), value=v.strip()))
            return real_query, filters
        return query_str, self.default_filters or MetadataFilters(filters=[])

    def query(self, query_bundle: QueryBundle) -> str:
        real_query, filters = self._parse_filters(query_bundle.query_str)
        # Rebuild query bundle with filters attached
        filtered_bundle = QueryBundle(query_str=real_query, custom_embedding_strs=[query_bundle.query_str])
        # Note: actual filter application depends on your vector store
        # This pattern works with Pinecone, Weaviate, Qdrant via their LlamaIndex integrations
        return self.base_engine.query(filtered_bundle)

# Usage in tool description:
tool = QueryEngineTool(
    query_engine=FilteredQueryEngine(policy_index.as_query_engine()),
    metadata=ToolMetadata(
        name="policy_docs",
        description=(
            "HR policies. Prefix query with filters: 'tenant:acme,version:2024 | your question'. "
            "Example: 'tenant:acme | parental leave policy'"
        ),
    ),
)

Tradeoff: this pushes filter syntax into the agent’s prompt. It works for a handful of known keys but doesn’t scale to dynamic schemas. For complex filtering, build a dedicated FunctionTool that accepts structured arguments and constructs the query engine internally.

Structured tool alternative: when you need typed arguments

QueryEngineTool only exposes input: str. If you need tenant_id: str, date_range: tuple[date, date], or tags: list[str], wrap the engine in a FunctionTool with a Pydantic schema.

from llama_index.core.tools import FunctionTool
from pydantic import BaseModel, Field
from datetime import date
from typing import Optional

class PolicyQuery(BaseModel):
    question: str = Field(description="Natural language question about policies")
    tenant_id: Optional[str] = Field(default=None, description="Tenant identifier for isolation")
    effective_date: Optional[date] = Field(default=None, description="Policy version as of this date")

def query_policies(question: str, tenant_id: Optional[str] = None, effective_date: Optional[date] = None) -> str:
    filters = MetadataFilters(filters=[])
    if tenant_id:
        filters.filters.append(ExactMatchFilter(key="tenant_id", value=tenant_id))
    if effective_date:
        filters.filters.append(ExactMatchFilter(key="effective_date", value=effective_date.isoformat()))
    
    engine = policy_index.as_query_engine(filters=filters, similarity_top_k=4)
    return str(engine.query(question))

policy_tool = FunctionTool.from_defaults(
    fn=query_policies,
    name="policy_docs",
    description="HR policies with optional tenant and date filtering.",
    fn_schema=PolicyQuery,
)

The agent now calls policy_docs(question="...", tenant_id="acme") with validated arguments. This is cleaner than string parsing and lets you add defaults, validation, and docs that appear in the agent’s function-calling prompt.

When to switch: as soon as you have more than one filter dimension or non-technical users who will read the tool schema. The string-convention approach is technical debt from day one.

Error handling and fallback behavior

Agents retry on tool errors, but the default exception surface is noisy. Wrap your engine to return structured error payloads the agent can reason about.

from llama_index.core.tools import FunctionTool
from llama_index.core.base.response.schema import Response

class ToolResult(BaseModel):
    success: bool
    content: str
    error: Optional[str] = None
    source_nodes: list = []

def safe_query(question: str) -> str:
    try:
        result = policy_engine.query(question)
        return ToolResult(success=True, content=str(result), source_nodes=result.source_nodes).model_dump_json()
    except Exception as e:
        return ToolResult(success=False, content="", error=f"{type(e).__name__}: {e}").model_dump_json()

safe_tool = FunctionTool.from_defaults(
    fn=safe_query,
    name="policy_docs",
    description="HR policies. Returns JSON with success flag, content, and error details.",
)

The agent sees valid JSON on every call. It can retry with a rephrased query on success: false instead of hallucinating around a stack trace. This pattern also works well when you route through an inference gateway that enforces per-model timeouts and fallback — the tool returns a consistent envelope regardless of which provider actually answered.

Evaluation: does the agent pick the right tool?

Don’t ship without a small eval set. Write 20-30 questions spanning each tool’s domain and the boundaries between them. Run the agent, log which tool fired, and measure precision/recall at the tool-selection level.

eval_cases = [
    ("What's the parental leave policy?", "policy_docs"),
    ("How do I roll back the payments service?", "runbooks"),
    ("What's the auth header for the user API?", "api_specs"),
    ("Deploy checklist for new microservice", "runbooks"),  # boundary case
]

def eval_tool_selection(agent, cases):
    correct = 0
    for question, expected_tool in cases:
        response = agent.chat(question)
        # Inspect the tool calls in the response
        tool_calls = getattr(response, "tool_calls", [])
        called = [tc.tool_name for tc in tool_calls]
        if expected_tool in called:
            correct += 1
        else:
            print(f"FAIL: '{question}' -> {called} (expected {expected_tool})")
    return correct / len(cases)

If precision is below 90%, tighten descriptions or add a router tool that classifies the query before retrieval. A simple FunctionTool with a few-shot prompt can route to the right QueryEngineTool and cut misroutes in half.

Caching and cost control

Every tool call hits the vector store and the LLM. For repeated questions — common in support bots — cache the tool output.

from functools import lru_cache
import hashlib

@lru_cache(maxsize=512)
def cached_query(cache_key: str, question: str) -> str:
    return str(policy_engine.query(question))

def query_with_cache(question: str) -> str:
    key = hashlib.md5(question.encode()).hexdigest()[:16]
    return cached_query(key, question)

Caveat: cache invalidation is hard when documents update. Either version your index namespace (policy_docs_v3) and include that in the cache key, or accept stale answers for a TTL window. For regulated domains, skip caching entirely and optimize the query engine instead (smaller top_k, response_mode="compact", cheaper synthesis model).

Putting it together: a production-ready pattern

# tools/policy_tool.py
from llama_index.core.tools import FunctionTool
from llama_index.core.vector_stores import MetadataFilters, ExactMatchFilter
from pydantic import BaseModel, Field
from datetime import date
from typing import Optional
import json

class PolicyQuery(BaseModel):
    question: str = Field(description="Natural language question about HR policies")
    tenant_id: Optional[str] = Field(default=None, description="Tenant identifier")
    effective_date: Optional[date] = Field(default=None, description="Policy version date")

def make_policy_tool(index, synthesis_llm):
    def query(question: str, tenant_id: Optional[str] = None, effective_date: Optional[date] = None) -> str:
        filters = MetadataFilters(filters=[])
        if tenant_id:
            filters.filters.append(ExactMatchFilter(key="tenant_id", value=tenant_id))
        if effective_date:
            filters.filters.append(ExactMatchFilter(key="effective_date", value=effective_date.isoformat()))
        
        engine = index.as_query_engine(
            filters=filters,
            similarity_top_k=4,
            response_mode="compact",
            llm=synthesis_llm,
        )
        result = engine.query(question)
        return json.dumps({
            "answer": str(result),
            "sources": [n.node_id for n in result.source_nodes],
            "tenant_id": tenant_id,
        })
    
    return FunctionTool.from_defaults(
        fn=query,
        name="policy_docs",
        description="HR policies: benefits, leave, compliance. Supports tenant_id and effective_date filters.",
        fn_schema=PolicyQuery,
    )

# agent/setup.py
from llama_index.core.agent import ReActAgent
from llama_index.llms.openai import OpenAI
from tools.policy_tool import make_policy_tool
from tools.runbook_tool import make_runbook_tool
from tools.spec_tool import make_spec_tool

llm = OpenAI(model="gpt-4o-mini", temperature=0)
synthesis_llm = OpenAI(model="gpt-4o-mini", temperature=0)  # separate for cost tracking

tools = [
    make_policy_tool(policy_index, synthesis_llm),
    make_runbook_tool(runbook_index, synthesis_llm),
    make_spec_tool(spec_index, synthesis_llm),
]

agent = ReActAgent.from_tools(
    tools,
    llm=llm,
    verbose=True,
    max_iterations=8,  # prevent runaway loops
)

This structure keeps tool logic testable, separates retrieval concerns from agent logic, and gives you typed interfaces that survive refactoring. The agent becomes a thin orchestrator — exactly where it should be.