Hybrid search and reranking: which LLM APIs help

Most RAG systems that ship to production replace naive vector search with a hybrid search reranking LLM API pattern: combine BM25 keyword hits with dense retrieval, then collapse the merged candidate set with a reranker. This guide gives an ordered path to build that pipeline and compares the APIs that actually earn their keep.

Why hybrid retrieval needs reranking

BM25 catches exact token matches and rare terms. Dense vectors catch semantic drift. Neither alone covers both. Merging top-k from each source leaves you with a mixed list whose scores are not comparable.

A reranker takes the query and each candidate document, then outputs a calibrated relevance score. That step is what makes the final context fed to the generator trustworthy. Skip it and you will watch your answer quality swing on whichever retriever happened to win that query.

What a reranking LLM API can do

A dedicated reranking API accepts a query and a list of texts, returns reordered indices with scores. It is not a chat model. Using GPT-4o to score 100 documents means 100 prompt round-trips or one bloated prompt—both are slow and expensive.

Dedicated models from Cohere, Voyage, and Jina are trained for pairwise or listwise relevance. They return results in 20–200ms for a few dozen docs. The hybrid search reranking LLM API you choose should be a purpose-built ranker, not a general chatbot wrapped in a scoring prompt.

Step 1: Build the two retrieval paths

Stand up a keyword index (Elasticsearch, OpenSearch, or Postgres FTS) and a vector index (FAISS, Pinecone, Weaviate). Query both in parallel.

from elasticsearch import Elasticsearch
import faiss

es = Elasticsearch("http://localhost:9200")
vec_index = faiss.read_index("docs.faiss")

def retrieve_hybrid(q_text, q_vec, k=50):
    kw = es.search(index="docs", query={"match": {"text": q_text}}, size=k)
    kw_ids = [h["_id"] for h in kw["hits"]["hits"]]
    _, vec_ids = vec_index.search(q_vec, k)
    # merge, dedupe, preserve order loosely
    seen, merged = set(), []
    for doc_id in kw_ids + list(vec_ids[0]):
        if doc_id not in seen:
            seen.add(doc_id)
            merged.append(doc_id)
    return merged[:k]

Keep candidate count under 100 before reranking. More than that wastes reranker tokens and latency.

Step 2: Pick a reranking LLM API

Options that work in practice:

• Cohere Rerank: rerank-english-v3.0 or multilingual v3. Handles 100 docs, returns scores. Best-documented API.
• Voyage rerank-2: Similar latency, strong on code and technical docs.
• Jina reranker-v2-base: Open-weight, self-hostable, no per-call fee if you run it.
• OpenAI: No dedicated rerank endpoint. You can embed query and docs and sort by cosine, but that is not true reranking. Prompting a chat model is viable only for <10 docs.

Avoid using a generative LLM as a reranker at scale. The tradeoff is simple: a 7B ranker tuned for relevance beats a 70B chat model asked to output a score, at 1/100th the cost.

Step 3: Call the reranker

Using Cohere’s client directly:

import cohere
co = cohere.Client("COHERE_KEY")
resp = co.rerank(
    query="How do I rotate API keys?",
    documents=[d["text"] for d in candidates],
    top_n=5,
    model="rerank-english-v3.0"
)
top_docs = [candidates[r.index] for r in resp.results]

Voyage’s call is nearly identical:

import voyageai
vo = voyageai.Client("VOYAGE_KEY")
resp = vo.rerank("How do I rotate API keys?", [d["text"] for d in candidates], model="rerank-2", top_k=5)

If you already route completions through a gateway, you can keep the client surface small. n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models and forwards to providers offering reranking, with automatic fallback when a provider is rate-limited and per-token usage metering. That removes the need to juggle separate SDKs for generation and ranking.

Step 4: Fuse, rerank, truncate

Run reciprocal rank fusion (RRF) if you want a pre-rerank sort, but it is optional. The reranker supersedes it.

def rrf(kw_ranks, vec_ranks, k=60):
    scores = {}
    for doc_id, rank in {**kw_ranks, **vec_ranks}.items():
        scores[doc_id] = scores.get(doc_id, 0) + 1/(k+rank)
    return sorted(scores, key=scores.get, reverse=True)[:50]

Pass the fused top 50 to the reranker. Truncate to the 5–8 docs the generator needs. Feeding 20 docs to the LLM burns context and distracts it.

Metadata filtering before rerank

Apply tenant, date, and permission filters before the rerank call. Reranking documents the user cannot see is pure waste.

candidates = [c for c in candidates if acl_check(user, c)]

Step 5: Evaluate offline before trusting

Label 50 real queries with relevant doc IDs. Measure recall@10 before rerank and after. If reranking does not lift recall, your retrieval is broken, not the ranker.

Track cost: Cohere charges per reranked doc; Voyage similar. A gateway’s per-token metering helps attribute spend across providers. Log the reranker scores alongside generation latency to spot regressions.

Common pitfalls and tradeoffs

Sending unfiltered candidates. Apply metadata filters before rerank. Reranking irrelevant-but-matching docs wastes money and can push good hits down.

Reranking with a chat model. A hybrid search reranking LLM API built on a generative model will cost 100x and add seconds. Use a ranker.

No fallback. If the reranker 429s, your pipeline stalls. Use a secondary ranker or degrade to RRF. Gateways with automatic fallback simplify this.

Ignoring max input length. Each reranker has a max doc size (e.g., 512 tokens). Chunk appropriately or you will silently truncate.

Over-reranking. Reranking 200 docs to show 3 is a latency tax. Cap candidates at 50–100.

Reference pipeline

1. User query → embed + keyword search.
2. Merge top 50 from each, dedupe.
3. Filter by metadata.
4. Call reranker, get top 5.
5. Inject into prompt via OpenAI-compatible chat completion.
6. Log scores for eval.

def rag(query, user):
    q_vec = embed(query)
    cands = retrieve_hybrid(query, q_vec, k=50)
    cands = [c for c in cands if acl_check(user, c)]
    ranked = rerank(query, cands, top_n=5)
    ctx = "\n".join(d["text"] for d in ranked)
    return llm_chat(f"Context:\n{ctx}\n\nQ: {query}")

The hybrid search reranking LLM API decision is not about which vendor is biggest, but which ranker fits your latency budget and doc profile. Purpose-built rankers win; gateways that unify access reduce integration tax. Build the retrieval merge first, measure, then add the reranker where it counts.