What is a vector database, explained simply

A vector database stores and queries high-dimensional vectors — numerical representations of text, images, or other data — using approximate nearest neighbor (ANN) search to find semantically similar items in milliseconds. Unlike traditional databases that match exact values or lexical tokens, vector databases enable similarity search over meaning, which is the foundation of retrieval-augmented generation (RAG), recommendation engines, and anomaly detection. If you’re building anything that needs “find me things like this,” you need a vector database.

How a vector database works

At ingestion time, your application passes raw data (documents, images, user events) through an embedding model — typically a transformer like text-embedding-3-large, bge-large-en-v1.5, or a domain-specific model — which outputs a fixed-length vector, usually 768 to 3072 dimensions. The vector database indexes these vectors using an ANN algorithm (HNSW, IVF, DiskANN, or a hybrid) so that queries can return the top-k nearest neighbors without scanning every vector.

At query time, the same embedding model encodes the user’s question or input into a query vector. The database traverses its index, computes distances (cosine, dot product, or Euclidean), and returns the nearest vectors along with their associated metadata and payloads. Most production systems then re-rank the top 50–100 candidates with a cross-encoder or LLM before returning final results.

# Minimal ingestion pipeline
from sentence_transformers import SentenceTransformer
import httpx

model = SentenceTransformer("BAAI/bge-large-en-v1.5")

documents = [
    {"id": "doc-1", "text": "n4n.ai routes requests across 240+ models with automatic fallback"},
    {"id": "doc-2", "text": "Vector databases enable semantic search over embeddings"},
    {"id": "doc-3", "text": "HNSW indexes provide fast approximate nearest neighbor search"},
]

vectors = model.encode([d["text"] for d in documents], normalize_embeddings=True)

# Upsert to your vector DB (pseudo-client)
for doc, vec in zip(documents, vectors):
    client.upsert(
        collection="docs",
        id=doc["id"],
        vector=vec.tolist(),
        payload={"text": doc["text"]}
    )

# Query pipeline
query = "How does n4n.ai handle provider failures?"
query_vec = model.encode([query], normalize_embeddings=True)[0]

results = client.search(
    collection="docs",
    query_vector=query_vec.tolist(),
    limit=5,
    with_payload=True
)

for r in results:
    print(f"{r.id}  score={r.score:.4f}  text={r.payload['text'][:80]}")

Why vector databases matter for LLM applications

LLMs have finite context windows and no long-term memory. A vector database acts as external memory: you chunk documents, embed them, store them, and retrieve only the relevant slices at inference time. This keeps token costs down, latency predictable, and hallucinations constrained to retrieved evidence.

The alternative — stuffing everything into the context window — fails on three axes: cost (quadratic attention), latency (prefill time), and accuracy (lost-in-the-middle). A well-tuned retrieval pipeline with a vector database typically reduces context tokens by 10–100x while improving answer quality.

Beyond RAG, vector databases power:

• Recommendation and personalization: User and item embeddings enable “users like you also liked” at scale
• Anomaly detection: Embeddings of logs, traces, or network flows cluster normal behavior; outliers surface as distant vectors
• Deduplication and clustering: Near-duplicate detection for data cleaning, content moderation, or corpus deduplication
• Multimodal search: CLIP-style embeddings let you search images with text and vice versa

Concrete example: RAG over technical documentation

Suppose you maintain API docs for a platform with 50,000 pages. A developer asks: “How do I configure automatic fallback when a provider hits rate limits?”

Without a vector database: You either send all 50k pages (impossible), maintain a brittle keyword index (misses synonyms), or fine-tune a model (expensive, stale).

With a vector database:

1. Chunk each page into ~500-token segments with 50-token overlap
2. Embed with text-embedding-3-large (3072 dims) or bge-large-en-v1.5 (1024 dims)
3. Store in a collection with metadata: {"product": "gateway", "version": "v2", "section": "routing"}
4. At query time, embed the question, search top-20, filter by product="gateway", re-rank with a cross-encoder, pass top-5 to the LLM

# Hybrid search: vector + metadata filter
results = client.search(
    collection="docs",
    query_vector=query_vec.tolist(),
    limit=20,
    filter={"must": [{"key": "product", "match": {"value": "gateway"}}]},
    with_payload=True
)

# Cross-encoder re-rank (pseudo)
from sentence_transformers import CrossEncoder
ranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, r.payload["text"]) for r in results]
scores = ranker.predict(pairs)
reranked = sorted(zip(results, scores), key=lambda x: x[1], reverse=True)[:5]

This pattern — embed, filter, re-rank — is the production standard. The vector database handles the ANN search and metadata filtering; the cross-encoder handles precision.

Index algorithms: what you actually need to know

HNSW (Hierarchical Navigable Small World) is the default for most managed services because it delivers sub-10ms p99 latency at 95%+ recall on million-vector scales — provided the index fits in RAM. IVF partitions vectors into clusters (via k-means) and searches only the nearest clusters; it scales further but requires tuning nprobe (clusters to scan) at query time. DiskANN keeps the graph on SSD with a small in-memory cache, trading latency for capacity.

Practical rule: Start with HNSW on a managed service. Move to DiskANN or IVF only when your vector count × dimension × 4 bytes exceeds your RAM budget.

Filtering and hybrid search

Real workloads almost always combine vector similarity with metadata filters: “find similar error logs from the last hour for service payments in region us-east-1.” A vector database that pushes filters into the ANN traversal (pre-filtering) outperforms one that retrieves 1000 vectors then filters in memory (post-filtering).

Pre-filtering requires the index to be filter-aware. HNSW implementations handle this by storing filterable attributes alongside vectors and skipping neighbors that don’t match during graph traversal. Some databases (e.g., Qdrant, Weaviate, Pinecone) support payload indexes (inverted indexes on metadata) that accelerate pre-filtering.

{
  "filter": {
    "must": [
      {"key": "service", "match": {"value": "payments"}},
      {"key": "region", "match": {"value": "us-east-1"}},
      {"key": "timestamp", "range": {"gte": "2024-01-15T00:00:00Z", "lt": "2024-01-16T00:00:00Z"}}
    ]
  }
}

Hybrid search (vector + keyword) is another common pattern: BM25 on the original text catches exact matches (error codes, function names) while vectors catch semantic matches. Most managed services now support this natively.

Common misconceptions

“Vector databases are just k-NN indexes”

An index is a data structure; a database adds persistence, CRUD, transactions, replication, backups, access control, and query planning. You can build on FAISS or Annoy directly, but you’ll rebuild the database layer within six months.

“Higher dimensions = better accuracy”

Diminishing returns hit hard past 1024–1536 dimensions for general-purpose embeddings. Larger vectors increase storage, memory, and latency linearly while recall gains plateau. Matryoshka embeddings (truncatable representations) let you store 3072-dim vectors but query at 256 dims for speed.

“Cosine similarity is the only metric”

Dot product equals cosine when vectors are normalized. Euclidean (L2) is equivalent to cosine on normalized vectors up to a monotonic transform. Choose based on your embedding model’s training objective: OpenAI and BGE models expect cosine; some contrastive models work better with dot product. Don’t mix.

“You need a specialized vector database”

PostgreSQL with pgvector, SQLite with sqlite-vec, and Elasticsearch with dense vector support all handle million-vector workloads. They trade some ANN-specific optimizations (graph traversal parallelism, filter-aware indexing) for operational simplicity. If you already run Postgres, pgvector gets you 80% of the way with zero new infrastructure.

“Embedding model choice doesn’t matter”

It matters more than the database. A mediocre vector database with bge-large-en-v1.5 beats a best-in-class database with text-embedding-ada-002 on most benchmarks. Evaluate models on your data with your queries before optimizing the index.

Choosing a vector database: a decision framework

Benchmark with your actual data and query distribution. Synthetic benchmarks (ANN-Benchmarks, VectorDBBench) correlate poorly with production workloads because they ignore filtering, concurrent writes, and re-ranking pipelines.

Operational concerns that bite teams

Index rebuilds: Adding a new embedding model or changing chunking strategy requires re-embedding and re-indexing. Design for this from day one: version your embeddings (model=v1, model=v2) and keep the raw text so you can backfill.

Write throughput: HNSW inserts are slower than reads because the graph must be updated. Batch upserts (100–1000 vectors per request) and use async clients. If you ingest millions of vectors nightly, consider a write-optimized path: bulk-load into a fresh index, then swap aliases.

Tenant isolation: Multi-tenant RAG needs per-tenant namespaces or collections. Shared collections with tenant filters work but leak latency variance across tenants. Separate collections per tenant (or per tenant tier) give predictable performance.

Consistency: Most vector databases are eventually consistent on reads after writes. If your application requires “read your writes” (user uploads a doc, immediately queries it), either use a database with strong consistency guarantees or poll until the vector appears.

Summary

A vector database is a specialized store for high-dimensional embeddings that answers “what is similar to this?” at scale. It replaces keyword search with semantic search, enabling RAG, recommendations, and anomaly detection. The core loop — embed, index, filter, search, re-rank — is stable across use cases. Choose your embedding model first, your index algorithm second, and your database third. Operational maturity (backups, monitoring, scaling) matters more than microbenchmarks on ANN recall.

If you’re routing LLM traffic across multiple providers and need consistent embedding endpoints, an inference gateway can normalize the embedding API surface — but that’s a separate concern from the vector database itself.