Cross-encoders vs bi-encoders for reranking

The cross-encoder vs bi-encoder decision shapes your entire retrieval pipeline. Cross-encoders score query-document pairs jointly for maximum relevance accuracy, while bi-encoders embed independently for sub-millisecond latency at scale. Most production systems need both — but knowing where to draw the line saves months of rework.

What each architecture actually does

A bi-encoder passes the query and each document through the same transformer independently, producing two fixed-length vectors. Relevance reduces to a dot product or cosine similarity in embedding space. This means you can pre-compute document embeddings offline, index them in FAISS, HNSW, or pgvector, and serve queries with a single forward pass.

# Bi-encoder inference pattern
query_vec = model.encode(query)           # 1 forward pass
doc_vecs = precomputed_index.search(query_vec, k=100)  # ANN search
scores = (query_vec @ doc_vecs.T).squeeze()  # dot product

A cross-encoder concatenates query and document with a separator token and runs a single forward pass through the transformer. The [CLS] token (or pooled output) feeds a regression head that outputs a relevance score. No pre-computation possible — every (query, document) pair requires a full forward pass.

# Cross-encoder inference pattern
pairs = [(query, doc) for doc in candidate_docs]
scores = model.predict(pairs)  # 1 forward pass PER pair
ranked = sorted(zip(candidate_docs, scores), key=lambda x: x[1], reverse=True)

This architectural difference cascades into every operational dimension.

Latency and throughput reality

Bi-encoders dominate on raw speed. A BERT-base bi-encoder on a T4 GPU encodes ~2,000 queries/second at batch size 32. The ANN lookup adds microseconds. You can serve thousands of QPS on modest hardware.

Cross-encoders are fundamentally slower. The same BERT-base cross-encoder scores ~50 pairs/second on a T4. Reranking 100 candidates adds ~2 seconds of pure compute latency. Batch inference helps — scoring 100 pairs in one batch takes ~200ms instead of 2s — but you still pay quadratic attention over the concatenated sequence.

The latency gap narrows with distillation. A 6-layer MiniLM cross-encoder scores ~200 pairs/second. But the bi-encoder equivalent runs ~10,000 QPS. The ratio holds: cross-encoders cost 50-100x more compute per query.

Quality and capability differences

Cross-encoders win on relevance quality because they model full query-document interaction. Self-attention lets every query token attend to every document token. This captures nuanced matching: negation (“not expensive”), compositionality (“red leather sofa” ≠ “red” + “leather” + “sofa”), and positional dependencies.

Bi-encoders compress each side into a fixed vector (typically 384-1024 dimensions) before any interaction occurs. The dot product can only express linear similarity in that compressed space. Information loss is inevitable — the “bottleneck problem.”

Empirically, cross-encoders gain 5-15 nDCG@10 points over strong bi-encoders on BEIR benchmarks. The gap widens on:

• Long documents where relevant spans are buried
• Queries requiring reasoning across multiple document segments
• Domain-specific terminology the bi-encoder hasn’t seen during training

But bi-encoders catch up significantly when you:

• Fine-tune on in-domain data with hard negatives
• Use larger models (E5-large, BGE-large, GTE-large)
• Apply late interaction (ColBERT) or multi-vector approaches

Operational ergonomics

Bi-encoders fit cleanly into existing vector databases. You index once, update incrementally, and scale horizontally with standard sharding. No model serving infrastructure beyond the embedding endpoint. Most teams already have this stack.

Cross-encoders require a dedicated reranking service. You need:

• A model server (Triton, TorchServe, vLLM, or custom FastAPI)
• Request batching logic to maximize GPU utilization
• Queue management for traffic spikes
• Cache invalidation strategy when documents update
• Horizontal scaling with model replication

# Minimal cross-encoder service skeleton
from fastapi import FastAPI
from pydantic import BaseModel
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

app = FastAPI()
model = AutoModelForSequenceClassification.from_pretrained("cross-encoder/ms-marco-MiniLM-L-6-v2")
tokenizer = AutoTokenizer.from_pretrained("cross-encoder/ms-marco-MiniLM-L-6-v2")

class RerankRequest(BaseModel):
    query: str
    documents: list[str]
    top_k: int = 10

@app.post("/rerank")
async def rerank(req: RerankRequest):
    pairs = [(req.query, doc) for doc in req.documents]
    inputs = tokenizer(pairs, padding=True, truncation=True, max_length=512, return_tensors="pt")
    with torch.no_grad():
        scores = model(**inputs).logits.squeeze(-1).tolist()
    ranked = sorted(zip(req.documents, scores), key=lambda x: x[1], reverse=True)
    return {"results": ranked[:req.top_k]}

Bi-encoders also simplify A/B testing. Swap the embedding model, re-index, done. Cross-encoder changes require redeploying the reranker service and validating latency SLAs.

Ecosystem and model availability

Bi-encoder options are abundant and mature. The MTEB leaderboard tracks dozens of models across 50+ tasks. Strong defaults exist for every language and domain:

• General: E5-large-v2, BGE-large-en-v1.5, GTE-large
• Multilingual: E5-multilingual, BGE-m3, Jina-embeddings-v3
• Code: CodeBERT, UnixCoder, Voyage-code-2
• Long context: E5-mistral-7b-instruct, GritLM-7B

Cross-encoder selection is narrower. Most production teams use:

• MS MARCO trained: cross-encoder/ms-marco-MiniLM-L-6-v2, cross-encoder/ms-marco-MiniLM-L-12-v2
• Distilled from larger teachers: cross-encoder/ms-marco-TinyBERT-L-6
• Domain-adapted: fine-tuned on proprietary click data

Fewer open-source checkpoints exist because training cross-encoders requires labeled pairs — expensive to collect. Bi-encoders train on weaker supervision (contrastive learning on query-document pairs, or even unsupervised via ICT).

Comparison table

Which to choose

Start with bi-encoder only when:

• Latency budget < 50 ms p99 — Cross-encoder reranking blows this budget unless you aggressively limit candidates (top-10) and batch heavily.
• Index updates are frequent — Document inserts/deletes every few minutes favor bi-encoders. Cross-encoders don’t care, but your bi-encoder index must stay fresh.
• Team lacks ML serving expertise — Operating a batched cross-encoder service with proper queuing, autoscaling, and observability is real engineering work.
• Multilingual or code search — Strong bi-encoders exist. Cross-encoder coverage is thin outside English web search.

Add cross-encoder reranking when:

• Quality gains justify cost — A/B test shows >5% conversion/relevance lift. Measure on your traffic, not BEIR.
• Candidate set is small — Reranking top-20 from a bi-encoder costs ~40 ms. Top-1000 costs ~2 seconds. Set a hard candidate ceiling.
• Long documents with sparse relevance — Legal, medical, technical docs where the answer lives in paragraph 47. Cross-encoders find needles; bi-encoders average haystacks.
• You have labeled data — Fine-tuning a cross-encoder on your click/log data yields disproportionate returns vs. bi-encoder fine-tuning.

Hybrid architecture (what most serious teams ship):

Query → Bi-encoder (top-100) → Cross-encoder (top-10) → Final ranking
         ~5 ms                      ~40 ms                  ~50 ms total

This captures 80-90% of the cross-encoder quality gain at 10% of the latency cost. The bi-encoder handles recall; the cross-encoder handles precision.

Advanced variants worth knowing:

ColBERT / late interaction — Bi-encoder that keeps token-level embeddings, scores via max-sim at query time. Quality approaches cross-encoder, latency sits between the two. Requires specialized index (Vespa, Milvus with ColBERT support, or custom).

Multi-vector bi-encoders — Encode document as multiple vectors (per passage, per sentence). Better recall for long docs. Higher storage and search cost.

LLM-as-reranker — Prompt a 7B-70B model to score relevance. Quality ceiling is higher but latency is 500ms-2s per candidate. Only viable for top-5 with heavy caching. Some teams use this for final ranking after cross-encoder.

The cross-encoder vs bi-encoder trade-off is fundamentally a recall-precision-latency triangle. Bi-encoders maximize recall per millisecond. Cross-encoders maximize precision per candidate. Production systems that matter use both, with a hard candidate budget for the cross-encoder stage. Start simple, measure on your data, add complexity only when the numbers demand it.