Choosing a vector database for your RAG pipeline

Choosing a vector database for rag is one of those decisions that feels reversible until you have 50 million vectors in production and your p99 latency spikes during a reindex. The market has consolidated around a few clear categories, but the tradeoffs between them are rarely documented in a way that maps to actual engineering constraints. This guide walks through an ordered evaluation path: define your access patterns first, then match them to the right architecture, then validate with a representative workload.

Start with your access patterns, not the vendor matrix

Before you compare Pinecone vs. Weaviate vs. Qdrant, write down the queries your application actually runs. RAG workloads typically fall into three buckets, and each demands different index structures:

Pure semantic search — top-k nearest neighbors on dense vectors, no metadata filtering. This is the simplest case and almost any vector database handles it well at modest scale.

Hybrid search — vector similarity combined with exact-match or range filters on metadata (tenant_id, document_type, timestamp). This is where most production RAG systems live, and where index choice matters most.

Filtered-then-ranked — apply a selective filter first (e.g., “only this user’s documents”), then run vector search on the reduced set. This pattern favors databases that can push filters into the index scan rather than post-filtering.

Write down your top 5 query shapes with expected selectivity. If 80% of your queries filter by tenant_id before vector search, that single fact eliminates half the market.

Choose your index architecture

Vector databases implement approximate nearest neighbor (ANN) search using one of three dominant index types. Each has distinct latency, recall, and build-time characteristics.

HNSW (Hierarchical Navigable Small World)

The default choice for most general-purpose workloads. HNSW builds a multi-layer graph where each layer is a navigable small world network. Search starts at the top layer and descends.

When it works well: Pure semantic search, hybrid search with low-cardinality filters, datasets that fit in memory. Recall is tunable via ef_search at query time.

When it struggles: High-cardinality metadata filters (filtering on user_id across millions of users), datasets larger than RAM, frequent updates/deletes (graph repair is expensive).

# Typical HNSW config for a 768-dim embedding model
index_params = {
    "M": 16,              # connections per node (8-32 typical)
    "ef_construction": 200,  # build-time search width
    "ef_search": 100      # query-time search width, tune for recall/latency
}

IVF (Inverted File Index) + PQ (Product Quantization)

IVF partitions the vector space into coarse centroids (k-means), then quantizes residuals with PQ. Search scans only the nearest centroids.

When it works well: Billions of vectors, disk-backed storage, workloads where you can afford lower recall (0.9-0.95) for massive compression. PQ enables 8-16x memory reduction.

When it struggles: Low-latency requirements (centroid lookup + PQ decode adds overhead), frequent inserts (requires periodic re-clustering), high-recall needs (>0.98).

# IVF-PQ config example (Faiss-style params)
index_params = {
    "nlist": 4096,        # number of centroids (sqrt(N) rule of thumb)
    "nprobe": 32,         # centroids to probe at query time
    "m": 16,              # PQ subquantizers (dim / m must be integer)
    "nbits": 8            # bits per subquantizer
}

DiskANN / Vamana (graph-on-SSD)

A graph index designed for SSD-resident storage. Uses a variant of HNSW optimized for sequential I/O and larger fanout.

When it works well: Datasets 10x-100x larger than RAM, cost-sensitive deployments where you’d rather pay for NVMe than DRAM.

When it struggles: Write-heavy workloads, low-latency requirements (<10ms p99), small datasets where RAM is cheaper than engineering complexity.

Evaluate filtering architecture

This is where most evaluations go wrong. Vendors benchmark pure vector search on clean datasets. Your production queries look like:

SELECT * FROM vectors
WHERE tenant_id = 'acme-corp'
  AND document_type IN ('contract', 'invoice')
  AND created_at > '2024-01-01'
ORDER BY embedding <-> query_vector
LIMIT 10

Three filtering architectures exist:

Post-filtering — run ANN search to get 100-1000 candidates, then apply metadata filters in memory. Simple to implement, but recall collapses when filter selectivity is low. If your filter matches 0.1% of vectors, you need to retrieve 100,000 candidates to find 100 matches — defeating the purpose of ANN.

Pre-filtering (bitmap/roaring) — maintain inverted indexes on metadata fields, intersect bitmaps to get candidate set, then run exact vector search on that subset. Works well for high-selectivity filters. Fails when filter cardinality is high (millions of tenant_ids) because bitmap intersection becomes the bottleneck.

Fused/index-aware filtering — the index itself understands metadata. HNSW graphs with metadata-aware routing, or IVF with per-centroid metadata histograms. This is the hardest to build but scales to high-cardinality filters.

Pitfall: Ask vendors for latency numbers on your filter selectivity, not their benchmark. A database that does 2ms p99 on pure vector search may do 500ms when filtering by a low-selectivity tenant_id.

Quantize deliberately, not by default

Most vector databases now default to automatic quantization (scalar, PQ, SQ). This saves memory but costs recall. The tradeoff is not linear.

# Example: measuring recall drop from quantization
from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer('all-MiniLM-L6-v2')
queries = model.encode(test_queries)
docs = model.encode(test_docs)

# Baseline: float32 exact search
baseline_recall = evaluate_recall(queries, docs, index_type="flat")

# With scalar quantization (int8)
sq_recall = evaluate_recall(queries, docs, index_type="sq")

# With product quantization (PQ 16x8)
pq_recall = evaluate_recall(queries, docs, index_type="pq", m=16, nbits=8)

print(f"Flat: {baseline_recall:.3f}, SQ: {sq_recall:.3f}, PQ: {pq_recall:.3f}")
# Typical output: Flat: 0.998, SQ: 0.985, PQ: 0.942

Rule of thumb: Scalar quantization (int8) typically costs 1-2% recall for 4x memory savings — almost always worth it. Product quantization costs 5-10% recall for 8-16x savings — only worth it if you’re memory-bound and can tolerate the recall hit. Never enable quantization without measuring recall on your embeddings and your query distribution.

Plan for updates and deletes

RAG pipelines are not read-only. Documents get updated, deleted, re-embedded when models change. Vector databases handle this differently:

Append-only + periodic rebuild (Pinecone, some managed services) — writes are fast, but you pay for full reindex periodically. Acceptable if updates are batched nightly.

In-place updates with tombstones (Qdrant, Weaviate, Milvus) — supports real-time updates, but fragmentation degrades search quality over time. Requires compaction/merge operations.

Log-structured merge (LSM) style (LanceDB, Chroma with certain backends) — writes go to mutable buffer, periodically flushed to immutable segments. Good write throughput, but search must check multiple segments.

Pitfall: If your pipeline re-embeds 10% of documents weekly (model upgrade, chunking strategy change), test the database’s update throughput at that volume. Many ANN indexes degrade sharply after 20-30% mutation without rebuild.

Scale dimensions independently

Your scaling bottlenecks will not be uniform. Separate them:

Practical approach: Start with a single-node deployment that handles your projected 12-month vector count. Validate query latency at that scale. Only then evaluate horizontal scaling. Most teams over-engineer sharding before they have a working single-node baseline.

Run a representative benchmark

Vendor benchmarks use random vectors. Your embeddings have structure (clustering, anisotropy) that affects ANN performance. Build a benchmark harness with your actual data:

# Minimal benchmark structure
import time
import numpy as np
from statistics import median

def benchmark_search(index, queries, k=10, n_runs=100):
    latencies = []
    recalls = []
    
    for q in queries[:n_runs]:
        start = time.perf_counter()
        results = index.search(q, k=k)
        latencies.append((time.perf_counter() - start) * 1000)  # ms
        
        # Compute recall against ground truth (brute force)
        gt = brute_force_search(q, k=k)
        recall = len(set(results) & set(gt)) / k
        recalls.append(recall)
    
    return {
        "p50_ms": median(latencies),
        "p99_ms": np.percentile(latencies, 99),
        "mean_recall": np.mean(recalls),
        "min_recall": np.min(recalls)
    }

# Run with your actual embedding model and query distribution
test_queries = load_production_query_log(sample=1000)
results = benchmark_search(index, test_queries)
print(f"p50: {results['p50_ms']:.1f}ms, p99: {results['p99_ms']:.1f}ms, recall: {results['mean_recall']:.3f}")

Test three scenarios: cold start (first query after deploy), steady state, and under write load (simulate your ingestion pipeline running concurrently). The cold-start numbers often surprise teams — some databases lazily load index segments on first access.

Operational realities

Managed vs. self-hosted

Managed services (Pinecone, Zilliz Cloud, Qdrant Cloud) remove operational burden but constrain index tuning. You typically cannot adjust HNSW M/ef_construction or IVF nlist/nprobe — the vendor picks defaults. If your workload needs non-standard tuning, you need self-hosted.

Self-hosted (Qdrant, Weaviate, Milvus, Chroma, LanceDB) gives full control but requires:

• Understanding index build parameters for your data distribution
• Capacity planning for memory/disk
• Upgrade/maintenance windows for index rebuilds
• Monitoring for index quality degradation (recall drift)

Multi-tenancy isolation

If you serve multiple customers, you need either:

• Physical isolation: separate collections/indexes per tenant. Simple, but doesn’t scale to thousands of tenants (metadata overhead, connection pooling).
• Logical isolation: single index with tenant_id filter + row-level security. Requires fused filtering architecture to maintain latency.

Pitfall: Logical isolation with post-filtering fails catastrophically when a large tenant’s queries starve small tenants’ results. Test with skewed tenant distributions (one tenant = 50% of vectors).

Embedding model migration

You will change embedding models. Plan for it:

• Store model version with each vector
• Support dual-write during migration (write to old and new collections)
• Run shadow evaluation before cutover
• Budget for full reindex (can take days at 100M+ vectors)

Decision checklist

Before committing, verify each item:

• Defined top 5 query shapes with filter selectivity
• Measured baseline recall/latency on representative data for 2-3 candidates
• Tested filter performance at production cardinality
• Validated update/delete throughput at projected mutation rate
• Quantified memory/disk cost at 3x projected scale
• Confirmed embedding model migration path
• Load-tested cold start and concurrent write scenarios
• Evaluated operational burden (team expertise, on-call rotation)

Common traps to avoid

Trap: Optimizing for the wrong metric. Optimizing p50 latency when your SLA is p99. Optimizing recall@10 when your reranker only sees top-50. Measure what your pipeline actually consumes.

Trap: Ignoring the reranker. Most production RAG pipelines use a two-stage retrieve-then-rerank architecture. The vector database only needs to deliver high recall at k=50-100; the cross-encoder handles precision. Don’t over-invest in vector index recall beyond what the reranker can use.

Trap: Vendor lock-in on proprietary formats. Some managed services export only proprietary snapshots. Ensure you can export vectors + metadata in a standard format (Parquet, numpy, JSONL) for migration.

Trap: Underestimating metadata index size. A Roaring bitmap on a high-cardinality field (user_id across 10M users) can exceed the vector index size. Profile this before committing.

Final recommendation

For most teams starting a new RAG pipeline in 2024:

1. Default to Qdrant (self-hosted) or Qdrant Cloud — HNSW with payload filtering, good update support, active development, reasonable operational complexity. Payload indexes handle hybrid search well.

2. If you need massive scale on a budget — LanceDB (embedded, columnar, LSM-style) or Milvus (distributed, multiple index types). Both support disk-backed indexes for 10x RAM capacity.

3. If you want zero ops and accept constraints — Pinecone or Zilliz Cloud. Fastest time-to-production, but least tuning flexibility.

4. If you’re already on a specific cloud — check the managed offering (AWS OpenSearch Serverless with k-NN, Azure AI Search, GCP Vertex AI Vector Search). Integration with existing IAM/VPC/logging often outweighs pure feature comparison.

The database is the easiest component to swap later if you abstract the interface. Define a clean repository layer, ship with a reasonable default, and optimize when you have production signal.