How to debug Weaviate query performance issues

Slow vector searches rarely stem from a single cause. Effective debugging Weaviate query performance starts with reproducible measurements, then narrows down to index configuration, resource limits, and query shape. This guide walks through a concrete workflow to isolate and fix latency regressions in production clusters.

Step 1: Reproduce the slow query with a baseline measurement

Capture the exact query and run it in isolation. A single vector search against a small dataset should return in single-digit milliseconds; if it doesn’t, you have a baseline problem, not a scaling problem.

Use the Python client to time the call precisely:

import time
import weaviate

client = weaviate.connect_to_local()
collection = client.collections.get("Article")

query_vec = [0.01] * 1536  # replace with real embedding
start = time.perf_counter()
result = collection.query.near_vector(
    vector=query_vec,
    limit=10,
)
elapsed_ms = (time.perf_counter() - start) * 1000
print(f"p50 baseline: {elapsed_ms:.1f}ms for {len(result.objects)} objects")
client.close()

Run this three times and record the median. If latency varies by more than 20%, the host is contended or the cache is cold. Warm the index by issuing the same query twice before measuring.

Verify success: You have a stable median latency number and the query vector is representative of production traffic.

Step 2: Enable telemetry and scrape metrics

When debugging Weaviate query performance, never trust anecdotal slowness. Weaviate exposes Prometheus metrics on /metrics. Scrape them directly:

curl -s http://localhost:8080/metrics | grep -E "weaviate_query_duration_seconds|weaviate_vector_index_operations"

Set LOG_LEVEL=debug in the Weaviate container environment to surface slow query plans. In Docker Compose:

environment:
  LOG_LEVEL: debug
  QUERY_DEFAULTS_LIMIT: 25

Restart the node. The debug log will show HNSW traversal depth and filter application order.

Verify success: weaviate_query_duration_seconds_bucket shows a clear histogram, and debug logs print per-query index stats.

Step 3: Inspect vector index configuration and query-time parameters

Pull the class schema and check the vectorIndexConfig:

curl -s http://localhost:8080/v1/schema/Article | jq '.vectorIndexConfig'

Typical output:

{
  "distance": "cosine",
  "ef": 128,
  "efConstruction": 128,
  "maxConnections": 64,
  "dynamicEfMin": 100,
  "dynamicEfMax": 500
}

ef is the query-time candidate list size. A high ef improves recall but linearly increases latency. Override it per query to test sensitivity:

{
  Get {
    Article(
      nearVector: { vector: [0.01, 0.02], ef: 64 }
    ) {
      title
    }
  }
}

If dropping ef to 32 cuts latency in half with acceptable recall, your default is too high.

Verify success: You can correlate ef values with latency and recall on a fixed test set.

Step 4: Check node resource saturation

Resource contention is the most overlooked aspect of debugging Weaviate query performance. Query the node status endpoint:

curl -s http://localhost:8080/v1/nodes | jq '.nodes[] | {name: .name, cpu: .status.cpu, mem: .status.memory, disk: .status.disk}'

Look for CPU near 100%, available memory below the index size, or disk I/O wait. HNSW indexes are memory-mapped; if the OS pages them out, every query becomes a random read from disk.

If the node is saturated, no index tweak will save you. Move to Step 7.

Verify success: Node stats show <70% CPU and free memory exceeding the vector index footprint.

Step 5: Profile filters and hybrid search overhead

Unindexed filters force a full scan before the vector search. Add a where filter and measure:

from weaviate.classes.query import Filter

start = time.perf_counter()
result = collection.query.near_vector(
    vector=query_vec,
    filters=Filter.by_property("published").greater_than("2023-01-01"),
    limit=10,
)
print(f"filtered: {(time.perf_counter()-start)*1000:.1f}ms")

If the filtered query is 10x slower, the property lacks an index. Add it via schema:

curl -X PUT http://localhost:8080/v1/schema/Article \
  -H "Content-Type: application/json" \
  -d '{"properties": [{"name": "published", "dataType": ["date"], "indexFilterable": true}]}'

For hybrid search, the BM25 pass runs alongside vectors. Disable it temporarily to see its cost:

{
  Get {
    Article(nearVector: {vector: [0.01]}) { title }
  }
}

Verify success: Filtered queries show linear cost relative to result size, not class size.

Step 6: Tune HNSW parameters and segment size

If recall is low but latency is fine, raise ef or efConstruction. If latency is high but recall is fine, lower ef or maxConnections. These require reindexing for efConstruction and maxConnections, but ef can be changed live as shown in Step 3.

To change build-time params, recreate the class:

from weaviate.classes.config import Configure

collection = client.collections.create(
    "Article",
    vector_index_config=Configure.VectorIndex.hnsw(
        ef=256, ef_construction=256, max_connections=48, distance="cosine"
    ),
)

Smaller maxConnections reduces memory but increases traversal depth. Test with a fixed recall benchmark.

Verify success: Recall@10 on a held-out set meets target (>0.95) at the lowest possible ef.

Step 7: Scale out or adjust replication

Scaling is the final lever in debugging Weaviate query performance. If a single node is saturated, increase replication factor so reads distribute:

curl -X PUT http://localhost:8080/v1/schema/Article \
  -H "Content-Type: application/json" \
  -d '{"replicationConfig": {"factor": 2}}'

For multi-node clusters, ensure query nodes are separate from write nodes if using Weaviate’s distributed mode. Horizontal sharding via shardingConfig splits data across nodes:

{
  "shardingConfig": {
    "virtualPerPhysical": 128,
    "desiredCount": 3
  }
}

Verify success: Adding a replica reduces p95 latency proportionally to read concurrency.

Step 8: Verify the fix with a load test

Wrap the baseline query in a loop with concurrency to simulate traffic:

import concurrent.futures, time, weaviate

client = weaviate.connect_to_local()
collection = client.collections.get("Article")

def q(_):
    s = time.perf_counter()
    collection.query.near_vector(vector=[0.01]*1536, limit=10)
    return time.perf_counter() - s

with concurrent.futures.ThreadPoolExecutor(max_workers=16) as ex:
    latencies = list(ex.map(q, range(1000)))

latencies.sort()
p95 = latencies[int(0.95*len(latencies)]*1000
print(f"p95 under load: {p95:.1f}ms")

Compare against your Step 1 baseline. A successful fix shows p95 under your SLO (e.g., 50ms) with zero 5xx errors in the Weaviate logs.

Verify success: Load test sustains target QPS with p95 latency within SLO for at least 10 minutes.

Debugging vector latency is iterative. Measure, change one variable, measure again. The workflow above keeps you honest about whether a tweak actually moved the needle.