How to track recall and precision in pgvector search

Most teams ship pgvector similarity search and assume the nearest neighbors are good enough. If you care about result quality, you need a repeatable process for pgvector recall and precision tracking against a labeled ground truth set. This guide walks through building that pipeline with PostgreSQL and a small Python harness you can run in CI or against a replica.

Step 1: Define a ground truth dataset

You cannot measure recall without knowing the correct answers. For vector search, the ground truth is the exact top-k neighbors computed without an approximate index. Build a table for your vectors and a table for evaluation queries.

CREATE TABLE items (
    id bigint PRIMARY KEY,
    embedding vector(1536)
);

-- Approximate index used in production
CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops);

CREATE TABLE eval_queries (
    id bigint PRIMARY KEY,
    query_embedding vector(1536)
);

Populate with real or synthetic data. For a quick baseline, generate random unit vectors:

import psycopg2, numpy as np

conn = psycopg2.connect("postgres://user:pass@localhost/db")
cur = conn.cursor()

dim = 1536
n_items = 10000
for i in range(n_items):
    vec = np.random.randn(dim)
    vec /= np.linalg.norm(vec)
    cur.execute("INSERT INTO items (id, embedding) VALUES (%s, %s)",
                (i, vec.tolist()))

for i in range(200):
    vec = np.random.randn(dim); vec /= np.linalg.norm(vec)
    cur.execute("INSERT INTO eval_queries (id, query_embedding) VALUES (%s, %s)",
                (i, vec.tolist()))
conn.commit()

Now compute exact neighbors. Force a sequential scan so the HNSW index is not used:

SET enable_indexscan = off;
SET enable_seqscan = on;

CREATE TABLE ground_truth AS
WITH ranked AS (
  SELECT q.id AS query_id, i.id AS item_id,
         row_number() OVER (PARTITION BY q.id ORDER BY i.embedding <=> q.query_embedding) AS rn
  FROM eval_queries q, items i
)
SELECT query_id, item_id FROM ranked WHERE rn <= 10;

For 10k items and 200 queries this cross join is cheap. At 1M items, run it in batches of queries or accept that exact evaluation is an offline job—pgvector recall and precision tracking does not need to run on every request, just often enough to catch regressions.

Step 2: Capture pgvector approximate search results

Re-enable the index and pull the same top-k through the approximate path. pgvector uses the index automatically when present and the planner chooses it.

SET enable_indexscan = on;

Retrieve predictions in Python so you can persist them:

def get_approx_topk(cur, query_vec, k=10):
    cur.execute(
        "SELECT id FROM items ORDER BY embedding <=> %s LIMIT %s",
        (query_vec, k)
    )
    return [r[0] for r in cur.fetchall()]

cur.execute("SELECT id, query_embedding FROM eval_queries")
pred_rows = []
for qid, qvec in cur.fetchall():
    pred = get_approx_topk(cur, qvec, 10)
    for rank, item_id in enumerate(pred, 1):
        pred_rows.append((qid, item_id, rank))

cur.executemany(
    "INSERT INTO pred_results (query_id, item_id, rank) VALUES (%s,%s,%s)",
    pred_rows
)
conn.commit()

Create the storage table first:

CREATE TABLE pred_results (
    query_id bigint,
    item_id bigint,
    rank int
);

Step 3: Calculate recall and precision

For pure nearest-neighbor retrieval where ground truth is the exact top-k, recall@k and precision@k are mathematically identical: both measure the fraction of true top-k items retrieved. The moment you introduce a relevance threshold (e.g., cosine distance < 0.2), precision@k becomes the fraction of your top-k that are truly relevant, while recall@k stays the fraction of all relevant items captured.

from collections import defaultdict

cur.execute("SELECT query_id, item_id FROM ground_truth")
gt = defaultdict(set)
for qid, iid in cur.fetchall():
    gt[qid].add(iid)

cur.execute("SELECT query_id, item_id FROM pred_results")
pred = defaultdict(list)
for qid, iid in cur.fetchall():
    pred[qid].append(iid)

k = 10
recalls, precisions = [], []
for qid in gt:
    truth = gt[qid]
    retrieved = set(pred[qid][:k])
    intersect = len(truth & retrieved)
    recalls.append(intersect / len(truth))
    precisions.append(intersect / k)

print(f"Recall@{k}: {sum(recalls)/len(recalls):.3f}")
print(f"Precision@{k}: {sum(precisions)/len(precisions):.3f}")

Tuning the index

Low recall usually means hnsw.ef_search (or ivfflat.probes) is too small. Bump it per session and re-run Step 2:

SET hnsw.ef_search = 40;  -- pgvector 0.5+ default is 20

This loop—change parameter, measure, compare—is the core of pgvector recall and precision tracking. Keep the ground truth static so movements reflect index behavior, not shifted labels.

Step 4: Persist metrics for trend analysis

One-off numbers lie. Store each evaluation run so you can spot regressions after data imports or Postgres upgrades.

CREATE TABLE recall_metrics (
    run_id serial PRIMARY KEY,
    run_ts timestamptz DEFAULT now(),
    k int,
    recall float,
    precision float,
    index_config jsonb
);

Insert a summary row after each batch:

cur.execute(
    "INSERT INTO recall_metrics (k, recall, precision, index_config) VALUES (%s, %s, %s, %s)",
    (k, sum(recalls)/len(recalls), sum(precisions)/len(precisions),
     '{"hnsw.ef_search": 40}')
)
conn.commit()

Query recent trends:

SELECT run_ts, k, recall, precision
FROM recall_metrics
ORDER BY run_ts DESC
LIMIT 20;

Add a retention rule if the table grows: drop rows older than 90 days or aggregate into daily stats.

Step 5: Automate continuous monitoring

Wrap Steps 2–4 in a script and run it from cron or a CI job against a production read replica. Alert when recall drops below an agreed tolerance:

MIN_RECALL = 0.95
avg_recall = sum(recalls)/len(recalls)
if avg_recall < MIN_RECALL:
    raise SystemExit(f"Recall degraded: {avg_recall:.3f}")

If you already run Prometheus, push the gauge:

from prometheus_client import Gauge, push_to_gateway
g = Gauge('pgvector_recall', 'Recall@k for vector search')
g.set(avg_recall)
push_to_gateway('pushgateway:9091', job='pgvector_eval', registry=g._registry)

Treat pgvector recall and precision tracking as a first-class observability signal. Plot it next to p99 latency so you can see the exact tradeoff when tightening ef_search.

Step 6: Verify the pipeline works

Success means the script runs clean and writes a metric row. Concrete checks:

1. SELECT count(*) FROM ground_truth; returns 200 * 10 rows (for 200 queries, k=10).
2. SELECT count(*) FROM pred_results; matches that shape.
3. Printed Recall@10 is between 0 and 1. On 10k random vectors with ef_search=40, it should exceed 0.9; with default 20 it may be lower but still >0.8.
4. SELECT * FROM recall_metrics ORDER BY run_ts DESC LIMIT 1; shows the new run with non-null values.

If those hold, you have a working feedback loop. Adjust k, distance threshold, or index parameters and watch the numbers move.

Scaling the ground truth computation

The cross join in Step 1 is O(queries × items). At 10M items, computing exact neighbors for 200 queries means 2B distance operations—still feasible offline with a single Python worker in minutes, but you should avoid doing it on the primary. Use pg_dump of a subset or run on a replica with max_parallel_workers raised. Alternatively, sample a random 100k item subset for evaluation; recall measured on a subset correlates strongly with full-set recall for evenly distributed embeddings.

Closing notes

pgvector is fast, but approximate indexes are a tradeoff. The only way to manage that tradeoff is disciplined pgvector recall and precision tracking. Build the ground truth once, automate the comparison, and keep the metrics next to your latency dashboards. When the next model change swells your embedding dimension, you will know exactly what it cost.