Cohere Rerank explained: how it improves search results

Cohere Rerank is a cross-encoder model that takes a query and a candidate document pair and outputs a relevance score, letting you reorder retrieval results with far higher precision than embedding similarity alone. Unlike bi-encoders that compress queries and documents into fixed vectors independently, the cross-encoder attends to both simultaneously, capturing fine-grained semantic interactions that vector dot products miss. This makes it the standard second stage in two-stage retrieval pipelines where recall comes first and precision comes second.

How cross-encoder reranking works

A bi-encoder (like text-embedding-3-large or Cohere’s own embed-english-v3.0) maps queries and documents to vectors in the same space. Similarity is a dot product or cosine between two pre-computed vectors. This scales — you index document vectors once and query vectors on the fly — but it loses information. The query “apple earnings” and document “Apple reported record revenue” get high similarity, but so does “Apple pie recipe” because “apple” dominates the vector.

A cross-encoder concatenates query and document with a separator token and runs full self-attention over the combined sequence. Every query token attends to every document token. The model learns interactions like “earnings” modifying “Apple” toward the company sense, not the fruit sense. The output head is a single scalar: relevance score.

# Conceptual cross-encoder forward pass
def cross_encoder_score(query: str, doc: str) -> float:
    # Tokenize with separator
    tokens = tokenizer(
        query, doc,
        truncation="only_second",  # truncate doc if too long
        max_length=512,
        return_tensors="pt"
    )
    # Full self-attention over [CLS] query [SEP] doc [SEP]
    logits = model(**tokens).logits  # shape: (1, 1)
    return logits.item()  # higher = more relevant

Cohere’s Rerank API wraps this. You send a query and up to 1000 documents (strings or pre-chunked texts). The service returns each document with a relevance_score (0–1) and index (original position). You sort by score descending.

curl -X POST https://api.cohere.ai/v1/rerank \
  -H "Authorization: Bearer $COHERE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "rerank-english-v3.0",
    "query": "Q3 2024 revenue guidance",
    "documents": [
      "Apple Q3 2024 earnings call transcript...",
      "Apple pie recipe with cinnamon...",
      "Microsoft Q3 2024 cloud revenue..."
    ],
    "top_n": 3,
    "return_documents": false
  }'

{
  "results": [
    {"index": 0, "relevance_score": 0.94},
    {"index": 2, "relevance_score": 0.31},
    {"index": 1, "relevance_score": 0.02}
  ],
  "meta": {"api_version": {"version": "1"}}
}

The top_n parameter lets you ask for only the top k results back, saving bandwidth when you’re reranking hundreds of candidates.

Why it matters: precision at k

Vector search optimizes for recall@k — getting relevant documents somewhere in the top 50 or 100. Rerank optimizes for precision@k — putting the most relevant documents in the top 3–5 that actually feed your LLM context window.

This distinction drives the standard two-stage architecture:

1. Retrieve — ANN index (HNSW, IVF, DiskANN) over bi-encoder embeddings. Fast, scales to millions of docs. Returns 50–200 candidates.
2. Rerank — Cross-encoder over the candidate set. Slower, O(n) in candidates, but n is small. Returns top 5–10 for the generator.

# Two-stage retrieval pipeline
async def retrieve_and_rerank(query: str, k: int = 5, candidate_pool: int = 100):
    # Stage 1: vector search (fast, high recall)
    candidates = await vector_db.search(
        query_embedding=embed(query),
        top_k=candidate_pool
    )
    # Stage 2: cross-encoder rerank (precise, low latency at small n)
    reranked = await cohere.rerank(
        query=query,
        documents=[c.text for c in candidates],
        top_n=k
    )
    # Map back to original candidates with metadata
    return [candidates[r.index] for r in reranked.results]

Empirically, this pattern lifts nDCG@10 by 15–30% over vector search alone on benchmarks like BEIR, TREC, and MS MARCO. The gain is largest on queries requiring reasoning — negation, temporal ordering, entity disambiguation — where embedding similarity is easily fooled.

Concrete example: legal clause retrieval

Imagine a contract analysis tool. A lawyer asks: “termination clause without cause for convenience”. Your corpus has 50,000 contract chunks.

Vector search top 5:

1. “Termination for cause: either party may terminate upon 30 days notice for material breach…” (score 0.82)
2. “Termination for convenience: either party may terminate without cause upon 60 days notice…” (score 0.79)
3. “Cause definition: material breach includes failure to pay, IP infringement…” (score 0.76)
4. “Force majeure termination…” (score 0.71)
5. “Termination without cause clause sample…” (score 0.68)

The vector model ranks “termination for cause” higher than “termination for convenience” because “cause” appears in the query and the document shares more token overlap. The lawyer gets the wrong clause first.

After Cohere Rerank (rerank-english-v3.0) on top 50:

1. “Termination for convenience: either party may terminate without cause upon 60 days notice…” (0.96)
2. “Termination without cause clause sample…” (0.89)
3. “Termination for cause: either party may terminate upon 30 days notice for material breach…” (0.34)
4. “Mutual termination without cause provisions…” (0.78)
5. “Convenience termination notice period requirements…” (0.71)

The cross-encoder attends to “without cause” modifying “termination” and correctly identifies the convenience clause as the target. The cause clause drops to rank 3 with a low score because the model sees the semantic contradiction.

Latency and cost trade-offs you’ll actually face

Rerank is not free. A cross-encoder forward pass on 512 tokens takes 50–150ms on GPU depending on model size and batch size. Cohere’s managed API adds network overhead. At 100 candidates, you’re looking at 200–500ms p95 latency before your LLM even starts generating.

Mitigation strategies:

1. Limit candidate pool — 50 candidates is often enough. Diminishing returns past 100.
2. Async pipeline — Fire rerank while streaming LLM tokens for the previous turn, or overlap with other retrievals.
3. Cascade rerank — Cheap heuristic (BM25, metadata filter) → vector search → rerank top 20 → final LLM.
4. Distill to bi-encoder — For ultra-low latency, fine-tune a bi-encoder on cross-encoder labels. You lose some precision but gain 100x speed.

# Cascade: metadata filter -> vector -> rerank
async def cascade_retrieve(query: str, filters: dict, final_k: int = 5):
    # Stage 0: structured filter (milliseconds)
    candidate_ids = await metadata_store.filter(filters)  # e.g., jurisdiction, date range
    # Stage 1: vector search over filtered set (10-50ms)
    vectors = await vector_db.search_by_ids(
        query_embedding=embed(query),
        ids=candidate_ids,
        top_k=50
    )
    # Stage 2: rerank (100-300ms)
    reranked = await cohere.rerank(
        query=query,
        documents=[v.text for v in vectors],
        top_n=final_k
    )
    return [vectors[r.index] for r in reranked.results]

Cost: Cohere charges per 1000 documents reranked. At ~$1/1M tokens (input), reranking 100 docs of 500 tokens each costs ~$0.0005 per query. Negligible compared to LLM generation, but it adds up at millions of queries/day. Batch your rerank calls — the API accepts up to 1000 documents per request.

Common misconceptions

“Rerank replaces vector search”

No. Rerank needs candidates. Vector search (or BM25, or hybrid) provides them. The cross-encoder is too slow to score your full corpus. Two-stage is the architecture; single-stage cross-encoder over millions of docs is not viable.

“Higher relevance_score means the answer is correct”

The score measures query-document relevance, not factual accuracy. A document can be highly relevant to the query and still contain hallucinations, outdated info, or the wrong jurisdiction. Rerank improves retrieval precision; verification remains a separate step.

“I should rerank everything the vector DB returns”

If your vector DB returns 200 candidates, reranking all 200 costs 4x more latency than reranking 50, with marginal precision gain. The top 50 from a good embedding model already contain the answer 95%+ of the time. Set candidate_pool based on your recall@k curve, not a fixed large number.

“Cohere Rerank only works with Cohere embeddings”

The API accepts raw text. It doesn’t know or care how you got the candidates. You can use OpenAI embeddings, bge-large, e5-mistral, BM25, or a hybrid fusion — rerank sits on top. The only coupling is tokenization: Cohere’s models expect their tokenizer’s conventions (byte-level BPE, specific special tokens). If you pre-chunk documents, keep chunks under 512 tokens to avoid truncation losing signal.

“Multilingual rerank is just translation”

rerank-multilingual-v3.0 is trained on 100+ languages jointly, not translate-then-rank. It learns cross-lingual relevance directly. In practice, it outperforms translate-then-rerank on low-resource languages because the model sees query-document pairs in the same script. But for high-resource languages (en, zh, es, fr, de), the English model on translated text is competitive. Test your language pair.

When to skip rerank

• Latency-critical paths where 200ms extra is unacceptable (real-time chat, autocomplete).
• High-recall regimes where you feed 50+ docs to a long-context LLM anyway — the generator does implicit reranking.
• Structured lookups where metadata filters + exact match already give perfect precision (e.g., “show me contract #12345”).
• Budget-constrained prototypes — vector-only gets you 80% of the way for 0% of the rerank cost.

Model versions and what changed

v3 models support 4096-token context (vs 512 for v2), meaning you can rerank full document sections without aggressive chunking. The trade-off: longer sequences = higher latency and cost. For most search use cases, 512-token chunks remain the sweet spot.

Integration checklist

• Set candidate_pool based on recall@50 of your vector index (measure it).
• Chunk documents to ≤512 tokens for v2, ≤4096 for v3 — but prefer smaller chunks for latency.
• Use top_n in the API request to avoid transferring full document text back.
• Cache rerank scores for repeated queries (query + doc_id hash as key).
• Monitor p95 latency; if it exceeds your SLA, reduce candidate pool or add cascade filters.
• Log relevance_score distributions per query type — bimodal distributions suggest the reranker is confused, often due to ambiguous queries.

Cohere Rerank explained simply: it’s a precision layer that sits on top of recall-optimized retrieval. The cross-encoder architecture captures query-document interactions that bi-encoders compress away. In production, it’s a 100–300ms latency spend for a measurable precision@k lift — worth it when the generator’s context window is small and answer quality matters.