BM25 explained: the algorithm behind keyword search

BM25 is a probabilistic ranking function that scores documents against a query using term frequency saturation, inverse document frequency, and document length normalization. It replaced TF-IDF as the default in Elasticsearch, OpenSearch, and Lucene because it handles real-world text distributions better. If you build search, you need to understand how its parameters shape relevance.

How BM25 works

The BM25 algorithm explained simply: it scores each document-query pair by summing contributions from each query term. The formula for a single term $t$ in document $d$ is:

$$\text{score}(d, t) = \text{IDF}(t) \cdot \frac{f(t, d) \cdot (k_1 + 1)}{f(t, d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl}})}$$

Where:

• $f(t, d)$ — raw term frequency in the document
• $|d|$ — document length (field length in tokens)
• $\text{avgdl}$ — average document length across the index
• $k_1$ — controls term frequency saturation (default 1.2)
• $b$ — controls document length normalization (default 0.75)

The IDF component uses the probabilistic variant:

$$\text{IDF}(t) = \ln \frac{N - n(t) + 0.5}{n(t) + 0.5} + 1$$

Where $N$ is total documents and $n(t)$ is documents containing term $t$. The $+0.5$ smoothing prevents division by zero and negative scores for terms appearing in more than half the corpus.

Term frequency saturation

Raw term frequency assumes more occurrences always means more relevant. BM25 disagrees. The fraction $\frac{f \cdot (k_1 + 1)}{f + k_1 \cdot \dots}$ saturates: each additional occurrence adds less score than the previous one.

def tf_saturation(tf, k1=1.2):
    return (tf * (k1 + 1)) / (tf + k1)

for tf in [1, 2, 3, 5, 10, 20, 50]:
    print(f"tf={tf:2d} -> score={tf_saturation(tf):.3f}")

Output:

tf= 1 -> score=1.000
tf= 2 -> score=1.455
tf= 3 -> score=1.714
tf= 5 -> score=2.000
tf=10 -> score=2.273
tf=20 -> score=2.462
tf=50 -> score=2.561

Going from 1 to 2 occurrences adds ~0.45 score. Going from 20 to 50 adds only ~0.10. This matches how humans judge relevance — the fifth “database” in a doc matters less than the first.

Tune $k_1$ higher (2.0–3.0) for verbose fields like body_text where repetition signals importance. Tune lower (0.5–1.0) for short fields like title or tags where a single occurrence is already strong signal.

Inverse document frequency

IDF downweights terms that appear everywhere. The probabilistic IDF formula has a useful property: terms in more than half the documents get negative IDF, which BM25 floors at zero via the $+1$ offset.

def bm25_idf(N, df):
    import math
    return math.log((N - df + 0.5) / (df + 0.5)) + 1

N = 1_000_000
for df in [1, 10, 100, 1_000, 10_000, 100_000, 500_000, 900_000]:
    print(f"df={df:7d} -> IDF={bm25_idf(N, df):.3f}")

Output:

df=       1 -> IDF=14.509
df=      10 -> IDF=12.206
df=     100 -> IDF=9.903
df=    1000 -> IDF=7.600
df=   10000 -> IDF=5.303
df=  100000 -> IDF=3.000
df=  500000 -> IDF=1.000
df=  900000 -> IDF=1.000

Rare terms (df=1) get massive weight. Common terms (df=500k) get minimum weight of 1.0. Terms in 90% of docs don’t go negative — they just contribute nothing discriminative.

Document length normalization

Long documents naturally contain more term occurrences. Without normalization, they’d dominate results. The factor $(1 - b + b \cdot \frac{|d|}{\text{avgdl}})$ scales the denominator:

• $b = 0$: no length normalization (raw TF)
• $b = 1$: full normalization (pivoted length)
• $b = 0.75$: default, partial normalization

def length_norm(doc_len, avgdl, b=0.75):
    return 1 - b + b * (doc_len / avgdl)

avgdl = 300
for doc_len in [50, 100, 200, 300, 500, 1000, 2000]:
    norm = length_norm(doc_len, avgdl)
    print(f"len={doc_len:4d} -> norm={norm:.3f} (denominator multiplier)")

Output:

len=  50 -> norm=0.375
len= 100 -> norm=0.500
len= 200 -> norm=0.750
len= 300 -> norm=1.000
len= 500 -> norm=1.250
len=1000 -> norm=2.000
len=2000 -> norm=3.500

A 50-token doc gets its TF multiplied by ~2.67x (denominator 0.375). A 2000-token doc gets divided by 3.5x. The default $b=0.75$ assumes longer docs are somewhat more likely to be relevant, but not proportionally.

Putting it together: a concrete example

Index three documents, query “vector database”:

Assume: $N=1000$, $\text{avgdl}=150$, $k_1=1.2$, $b=0.75$. Term stats: “vector” df=50, “database” df=200.

import math

N = 1000
avgdl = 150
k1 = 1.2
b = 0.75

df = {"vector": 50, "database": 200}

def idf(term):
    return math.log((N - df[term] + 0.5) / (df[term] + 0.5)) + 1

def score_doc(doc_tf, doc_len):
    total = 0.0
    for term, tf in doc_tf.items():
        idf_val = idf(term)
        norm = 1 - b + b * (doc_len / avgdl)
        tf_component = (tf * (k1 + 1)) / (tf + k1 * norm)
        total += idf_val * tf_component
    return total

# Term frequencies per doc
doc_A = {"vector": 3, "database": 4}
doc_B = {"vector": 2, "database": 3}
doc_C = {"vector": 0, "database": 1}

for name, tf, length in [("A", doc_A, 120), ("B", doc_B, 80), ("C", doc_C, 200)]:
    s = score_doc(tf, length)
    print(f"Doc {name}: score={s:.3f}")

Output:

Doc A: score=18.247
Doc B: score=16.891
Doc C: score=1.000

Doc A wins: higher term frequencies, reasonable length. Doc B is shorter so gets a length boost, but lower TF hurts. Doc C only matches “database” once in a long doc — minimal score.

Why BM25 matters for engineers

It’s the baseline you beat

Every semantic search demo compares against BM25. If your vector search doesn’t outperform a tuned BM25 baseline on keyword-heavy queries (error codes, product SKUs, exact phrases), you’re adding complexity for negative ROI. Run the baseline first.

It handles hybrid search correctly

Modern search stacks combine BM25 and vector scores. The standard approach: reciprocal rank fusion (RRF) or weighted sum.

def rrf(rank_lists, k=60):
    """Reciprocal Rank Fusion across multiple rankers."""
    scores = {}
    for rank_list in rank_lists:
        for rank, doc_id in enumerate(rank_list, 1):
            scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

# Example: BM25 ranks [A, B, C], vector ranks [B, A, D]
bm25_ranks = ["A", "B", "C"]
vector_ranks = ["B", "A", "D"]
fused = rrf([bm25_ranks, vector_ranks])
print(fused)  # [('A', 0.032), ('B', 0.032), ('C', 0.016), ('D', 0.016)]

RRF needs no score calibration — it only needs ranked lists. This is why BM25 stays relevant even in vector-first architectures.

It’s interpretable

When a stakeholder asks “why is this result first?”, you can trace the score components: “Document A has 4 occurrences of ‘database’ (IDF 5.3) in a 120-token field (length norm 0.9), contributing 18.2 points.” Try explaining a 768-dimensional dot product to a product manager.

Common misconceptions

“BM25 is just TF-IDF with extra steps”

TF-IDF uses linear TF and logarithmic IDF without length normalization. BM25 adds saturation, probabilistic IDF, and pivoted length normalization. The differences matter: on TREC benchmarks, BM25 consistently beats TF-IDF by 10–25% MAP. The parameters $k_1$ and $b$ give you knobs TF-IDF lacks.

“Default parameters work for everything”

Defaults ($k_1=1.2, b=0.75$) are reasonable for general web search. They’re often wrong for:

• Short fields (titles, tags): lower $k_1$ (0.3–0.8), lower $b$ (0.2–0.5)
• Long fields (full text, logs): higher $k_1$ (1.5–2.5), higher $b$ (0.8–1.0)
• Structured fields (categories, IDs): $b=0$ (no length norm), $k_1 \to \infty$ (binary presence)

Tune per field. In Elasticsearch, set similarity per field mapping:

PUT /products
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "similarity": "bm25_title"
      },
      "description": {
        "type": "text",
        "similarity": "bm25_body"
      }
    }
  },
  "settings": {
    "index": {
      "similarity": {
        "bm25_title": { "type": "BM25", "k1": 0.5, "b": 0.3 },
        "bm25_body": { "type": "BM25", "k1": 1.8, "b": 0.8 }
      }
    }
  }
}

“BM25 doesn’t work for short queries”

Short queries (1–2 terms) are where BM25 shines. The IDF component does heavy lifting. For “error 504”, the rarity of “504” dominates. Vector search struggles here because short queries produce unstable embeddings. Hybrid search exists partly because BM25 covers the short-query case better than dense vectors.

“You need BM25F for field-weighted search”

BM25F extends BM25 to combine multiple fields with different weights and length normalizations. Elasticsearch’s multi_match with type: most_fields or cross_fields approximates this. True BM25F is rarely necessary — per-field similarities plus copy_to a catch-all field handles 95% of cases with less complexity.

Tuning checklist

When relevance feels off, check these in order:

1. Analyze term statistics — run _termvectors or explain API to see actual TF, DF, field lengths
2. Verify analyzer consistency — query and index must use the same tokenization, or term frequencies lie
3. Adjust $k_1$ per field — short fields need lower saturation
4. Adjust $b$ per field — long fields need stronger length penalty
5. Check for stopword removal — removing “the” from index but not query breaks IDF math
6. Consider query-time boosting — ^2 on title field often beats tuning $k_1/b$

# Debug a specific query in Elasticsearch
GET /products/_search
{
  "query": { "match": { "title": "vector database" }},
  "explain": true
}

The explain output shows each term’s IDF, TF, length norm, and final contribution. Use it.

When to reach for something else

BM25 fails when:

• Synonymy matters — “car” vs “automobile” have zero lexical overlap
• Intent is ambiguous — “apple” could be fruit or company; context needed
• Natural language queries — “how do I fix a 504 error” has low keyword overlap with “nginx upstream timeout configuration”

That’s where vector search, cross-encoders, or LLM-based reranking enter. But they complement BM25 — they don’t replace it. The best search systems run BM25 first, then rerank top-K with something smarter.