Hybrid search query engine in LlamaIndex tutorial

This llamaindex hybrid search query engine tutorial walks you through building a retrieval system that combines dense vector search with sparse BM25 keyword matching. Pure vector search misses exact terminology and rare tokens; pure keyword search misses semantic similarity. Hybrid search gives you both, and LlamaIndex makes it straightforward to wire up.

Prerequisites

You need Python 3.10+ and an OpenAI API key (or any LlamaIndex-compatible embedding model). Install the core packages:

pip install llama-index llama-index-vector-stores-chroma llama-index-retrievers-bm25 chromadb

If you prefer a local embedding model, swap OpenAIEmbedding for HuggingFaceEmbedding and install sentence-transformers. The code below assumes OpenAI for brevity.

You also need a ChromaDB instance. For local development, the in-process client works fine:

import chromadb
chroma_client = chromadb.Client()  # in-memory; use PersistentClient(path="...") for disk

Why hybrid search matters

Vector embeddings excel at “find me documents about X” but struggle with “find me documents containing the exact phrase ‘Section 4.2(b)’ or the error code ‘ERR_CONNECTION_RESET’.” BM25 excels at the latter. A hybrid retriever runs both, merges the results, and re-ranks them — typically with reciprocal rank fusion (RRF) or a learned cross-encoder.

LlamaIndex provides QueryFusionRetriever for exactly this pattern. It handles the parallel execution, deduplication, and fusion logic so you don’t have to roll your own.

Prepare your documents

Load and chunk your corpus. The chunk size and overlap affect both retrievers differently: smaller chunks help BM25 precision, larger chunks help vector recall. Start with 512 tokens and 50 overlap.

from llama_index.core import SimpleDirectoryReader, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=50)

documents = SimpleDirectoryReader("./data").load_data()
nodes = Settings.node_parser.get_nodes_from_documents(documents)
print(f"Loaded {len(documents)} docs, split into {len(nodes)} nodes")

Expected output:

Loaded 47 docs, split into 312 nodes

Build the vector index

Use Chroma as the vector store. This persists to disk so you can reuse it across runs.

from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.chroma import ChromaVectorStore

chroma_collection = chroma_client.get_or_create_collection("hybrid_demo")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

vector_index = VectorStoreIndex(nodes, storage_context=storage_context, show_progress=True)
vector_index.storage_context.persist(persist_dir="./chroma_db")

The persist_dir is optional with Chroma since it manages its own persistence, but keeping the LlamaIndex metadata alongside it simplifies reloads.

Build the BM25 index

LlamaIndex’s BM25Retriever builds an in-memory index from nodes. For larger corpora, consider BM25Retriever.from_defaults with a persisted pickle or use a dedicated search engine like Tantivy or Elasticsearch. For this tutorial, in-memory is fine up to ~100k nodes.

from llama_index.retrievers.bm25 import BM25Retriever
import pickle
import os

bm25_path = "./bm25_index.pkl"

if os.path.exists(bm25_path):
    with open(bm25_path, "rb") as f:
        bm25_retriever = pickle.load(f)
else:
    bm25_retriever = BM25Retriever.from_defaults(
        nodes=nodes,
        similarity_top_k=10,
        stemmer="english",  # requires nltk; pip install nltk && python -m nltk.downloader punkt
        language="english",
    )
    with open(bm25_path, "wb") as f:
        pickle.dump(bm25_retriever, f)

print(f"BM25 index ready with {len(bm25_retriever.corpus)} documents")

Create the hybrid query engine

Now fuse the two retrievers. QueryFusionRetriever runs both in parallel, merges by RRF by default, and returns the top-k fused results. You can tune num_queries (for query rewriting), mode (“reciprocal_rerank”, “relative_score”, “simple”), and use_async.

from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.postprocessor import SimilarityPostprocessor

vector_retriever = vector_index.as_retriever(similarity_top_k=10)

fusion_retriever = QueryFusionRetriever(
    retrievers=[vector_retriever, bm25_retriever],
    similarity_top_k=10,
    num_queries=1,           # set >1 to enable query rewriting (requires LLM)
    mode="reciprocal_rerank",  # RRF fusion
    use_async=True,
    verbose=True,
)

# Optional: filter out very low-score results after fusion
fusion_retriever = SimilarityPostprocessor(fusion_retriever, similarity_cutoff=0.3)

query_engine = RetrieverQueryEngine.from_args(fusion_retriever)

Test it

Run a few queries that exercise both semantic and exact-match behavior.

test_queries = [
    "How do I configure authentication in the API gateway?",
    "ERR_CONNECTION_TIMEOUT error code meaning",
    "What are the rate limits for the payments endpoint?",
    "Section 4.2(b) compliance requirements",
]

for q in test_queries:
    print(f"\n=== Query: {q} ===")
    response = query_engine.query(q)
    print(f"Answer: {response.response[:300]}...")
    print(f"Sources: {len(response.source_nodes)} nodes retrieved")
    for i, node in enumerate(response.source_nodes[:3]):
        print(f"  [{i}] score={node.score:.3f} text={node.text[:120]}...")

Expected output (truncated):

=== Query: How do I configure authentication in the API gateway? ===
Answer: To configure authentication in the API gateway, you need to set the auth_provider...
Sources: 7 nodes retrieved
  [0] score=0.823 text=The API gateway supports multiple authentication providers including OAuth2, JWT, and API keys...
  [1] score=0.791 text=Authentication configuration is defined in the gateway.yaml file under the auth section...
  [2] score=0.745 text=For JWT validation, specify the jwks_uri and expected audience in the config...

=== Query: ERR_CONNECTION_TIMEOUT error code meaning ===
Answer: ERR_CONNECTION_TIMEOUT indicates that the client failed to establish a connection...
Sources: 5 nodes retrieved
  [0] score=0.887 text=Error code ERR_CONNECTION_TIMEOUT: The connection attempt timed out after 30 seconds...
  [1] score=0.654 text=Common causes include network latency, firewall rules, or the upstream service being unavailable...

Notice how the exact error code query pulls the precise definition (BM25 strength) while the conceptual question pulls relevant conceptual chunks (vector strength).

Tune the fusion weights

RRF treats both retrievers equally. If your domain favors one signal, weight them. QueryFusionRetriever accepts retriever_weights as a list matching the retriever order.

# Favor vector search 60/40 for semantic-heavy domains
fusion_retriever = QueryFusionRetriever(
    retrievers=[vector_retriever, bm25_retriever],
    similarity_top_k=10,
    num_queries=1,
    mode="reciprocal_rerank",
    retriever_weights=[0.6, 0.4],  # vector, bm25
    use_async=True,
)

For keyword-heavy domains (logs, legal, code), flip it to [0.3, 0.7]. Evaluate with a small labeled set: pick 20-30 representative queries, grade retrieved chunks for relevance, and grid-search the weight.

Add a cross-encoder reranker (optional but recommended)

Fusion retrieves candidates; a cross-encoder reranks them with full query-document attention. This is the single highest-ROI improvement for hybrid search.

from llama_index.postprocessor.cohere_rerank import CohereRerank
# or: from llama_index.postprocessor.flag_embedding_reranker import FlagEmbeddingReranker

# Cohere (API)
reranker = CohereRerank(top_n=5, model="rerank-english-v3.0")

# Or local (requires sentence-transformers and flag-embedding)
# reranker = FlagEmbeddingReranker(top_n=5, model="BAAI/bge-reranker-v2-m3")

query_engine = RetrieverQueryEngine.from_args(
    fusion_retriever,
    node_postprocessors=[reranker],
)

The reranker runs after fusion, so it sees the merged candidate set. Keep top_n small (3-5) for latency; the fusion similarity_top_k can stay larger (10-20) to give the reranker room to work.

Persist and reload in production

Don’t rebuild indexes on every startup. Persist both indexes and reload.

# Persist vector index (Chroma handles its own persistence)
vector_index.storage_context.persist(persist_dir="./chroma_db")

# Persist BM25 index
with open("./bm25_index.pkl", "wb") as f:
    pickle.dump(bm25_retriever, f)

# Reload on startup
def load_hybrid_engine():
    # Vector
    chroma_collection = chroma_client.get_collection("hybrid_demo")
    vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
    storage_context = StorageContext.from_defaults(vector_store=vector_store, persist_dir="./chroma_db")
    vector_index = VectorStoreIndex.from_vector_store(vector_store, storage_context=storage_context)
    vector_retriever = vector_index.as_retriever(similarity_top_k=10)

    # BM25
    with open("./bm25_index.pkl", "rb") as f:
        bm25_retriever = pickle.load(f)

    # Fusion
    fusion = QueryFusionRetriever(
        retrievers=[vector_retriever, bm25_retriever],
        similarity_top_k=10,
        mode="reciprocal_rerank",
        retriever_weights=[0.5, 0.5],
        use_async=True,
    )
    return RetrieverQueryEngine.from_args(fusion, node_postprocessors=[reranker])

Evaluate systematically

Ad-hoc testing isn’t enough. Build a small eval set: 30-50 queries with expected relevant doc IDs or passages. Use LlamaIndex’s RetrieverEvaluator with hit_rate and mrr metrics.

from llama_index.core.evaluation import RetrieverEvaluator

eval_queries = [
    {"query": "ERR_CONNECTION_TIMEOUT meaning", "expected_ids": ["node_42", "node_43"]},
    {"query": "authentication config gateway", "expected_ids": ["node_12", "node_19", "node_31"]},
    # ... more
]

evaluator = RetrieverEvaluator.from_metric_names(
    ["hit_rate", "mrr"],
    retriever=fusion_retriever,
)
results = await evaluator.aevaluate_dataset(eval_queries)
print(f"Hit rate: {results.metric_vals_dict['hit_rate']:.3f}")
print(f"MRR: {results.metric_vals_dict['mrr']:.3f}")

Target >0.7 hit rate and >0.5 MRR for a solid baseline. If you’re below, diagnose: check chunking, embedding model, BM25 tokenization, and fusion weights in that order.

Common pitfalls

Chunking mismatch. Vector search wants context; BM25 wants precision. If your chunks are 2000 tokens, BM25 drowns in noise. If they’re 100 tokens, vectors lose semantic coherence. Consider dual chunking: small chunks for BM25, larger parent chunks for vector, linked via node.ref_doc_id or a custom metadata field.

Stemming language. The stemmer="english" default hurts code, logs, and technical identifiers. For those corpora, disable stemming (stemmer=None) or use a custom tokenizer that preserves underscores, dots, and error codes.

Async event loop. use_async=True requires a running event loop. In scripts, wrap with asyncio.run() or call from an async context. In FastAPI/Starlette, it works natively.

Provider fallback. If you route embeddings through a gateway like n4n.ai, the same endpoint serves multiple providers. When one provider degrades, the gateway fails over transparently — your retriever code doesn’t change, but you avoid the “embedding service down” outage that kills both vector search and any LLM-based query rewriting.

Next steps

• Swap Chroma for a managed vector store (Pinecone, Weaviate, Qdrant) when you exceed single-node scale.
• Replace BM25 with a learned sparse encoder (SPLADE) for better semantic coverage in the sparse channel.
• Add query rewriting (num_queries=3-5) for complex multi-hop questions.
• Log every query, retrieved node IDs, fusion scores, and final answer for offline analysis and A/B testing.

Hybrid search isn’t a silver bullet, but it’s the minimal viable retrieval stack for any production RAG system. The code above gets you there in ~100 lines.