Monitoring Qdrant memory usage under high query load

Qdrant memory usage monitoring becomes critical the moment your vector search cluster faces sustained query bursts. A node that quietly climbs in RSS until the OOM killer strikes is a common failure mode that monitoring can catch early. This guide walks through a concrete setup to observe, alert, and validate memory behavior under load.

Step 1: Expose Qdrant metrics and confirm the endpoint

Qdrant exposes a Prometheus-compatible /metrics endpoint on its HTTP port (default 6333). The official Docker image serves it with no extra configuration.

Start a node locally to follow along:

docker run -p 6333:6333 -p 6334:6334 qdrant/qdrant:latest

Confirm the metrics surface is live:

curl -s http://localhost:6333/metrics | grep -E "process_resident_memory_bytes|qdrant_storage_ram_usage_bytes" | head

You should see lines beginning with those metric names. If the scrape returns nothing, check that you are not hitting the gRPC port (6334) or a reverse proxy that strips /metrics.

Step 2: Scrape metrics with Prometheus

Drop a minimal prometheus.yml next to your Qdrant instance:

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: qdrant
    static_configs:
      - targets: ['localhost:6333']

Launch Prometheus:

docker run -p 9090:9090 -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus:latest

Within a minute, the qdrant target should show as UP under http://localhost:9090/targets. Solid Qdrant memory usage monitoring starts with this scrape landing reliably. If you run in Kubernetes, use a ServiceMonitor instead of a static target.

Step 3: Identify the memory metrics that matter

Not every memory line in the export is useful for capacity planning. Focus on three:

process_resident_memory_bytes

The OS-level RSS of the Qdrant process. This is what the kernel counts against your cgroup limit and what the OOM killer watches.

qdrant_storage_ram_usage_bytes

Vector data and HNSW graphs explicitly loaded into RAM (not mmap’d). If you set on_disk: true for vectors, this stays low.

qdrant_filterable_ram_usage_bytes

Payload indices held in memory for filtering. Grows with the number of indexed fields, not query volume.

A quick PromQL check for current RSS:

process_resident_memory_bytes{job="qdrant"}

To see the rate of growth over a 5-minute window under load:

deriv(process_resident_memory_bytes{job="qdrant"}[5m])

Positive, non-zero derivatives during query spikes are the earliest signal of trouble. Also track process_virtual_memory_bytes; Qdrant mmap’s vector files, so virtual size can be multiples of physical RAM while RSS stays bounded.

HNSW graphs dominate steady-state RAM when vectors are in-memory. A rough upper bound is N * (dim*4 + m*8) bytes, where m is the graph degree. That estimate helps you spot abnormal growth versus expected scaling.

Step 4: Generate representative high query load

Monitoring a quiet node proves nothing. You need a load generator that issues concurrent similarity searches. The script below uses the official qdrant-client to fire parallel queries against a collection named docs.

First, create the collection with on-disk vectors to isolate query-time allocations:

from qdrant_client import QdrantClient

client = QdrantClient("localhost", port=6333)
client.recreate_collection(
    collection_name="docs",
    vectors_config={"size": 768, "distance": "Cosine", "on_disk": True},
)

Now the load test with bounded concurrency and timing:

import asyncio, time
from qdrant_client import QdrantClient

client = QdrantClient("localhost", port=6333)
CONCURRENCY = 50

async def search_once(i: int):
    vec = [0.01 * (i % 100) for _ in range(768)]
    await client.search_async(
        collection_name="docs",
        query_vector=vec,
        limit=10,
        with_payload=False,
    )

async def main():
    start = time.monotonic()
    tasks = [search_once(i) for i in range(CONCURRENCY)]
    await asyncio.gather(*tasks)
    print(f"{CONCURRENCY} searches in {time.monotonic()-start:.2f}s")

if __name__ == "__main__":
    asyncio.run(main())

Run it in a shell loop to sustain pressure:

while true; do python load.py; done

Confirm queries actually land by graphing qdrant_search_count{job="qdrant"}. If that counter is flat, your loader is misconfigured or pointing at the wrong host.

Step 5: Set alerting thresholds based on observed growth

Blind dashboards are not enough. Define an alert that fires before the cgroup limit hits. Assume your container has a 4 GiB memory cap.

Create rules.yml:

groups:
  - name: qdrant-memory
    rules:
      - alert: QdrantMemoryCreep
        expr: process_resident_memory_bytes{job="qdrant"} > 3.2e9
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Qdrant RSS above 80% of 4GiB limit"

Load it into Prometheus via --rules.file or a ConfigMap in Kubernetes. The threshold should be derived from your actual node memory limit, not guessed. Effective Qdrant memory usage monitoring pairs this alert with the derivative query from Step 3. In a multi-node cluster, scope the alert per instance:

process_resident_memory_bytes{job="qdrant"} > 3.2e9

and group by instance in Alertmanager so one noisy neighbor does not page the whole team.

Step 6: Verify your monitoring catches memory creep

Verification is concrete: apply load, confirm the graph moves, confirm the alert fires, then stop load and confirm recovery.

1. Start the Python loader in a background loop: while true; do python load.py; done
2. Open Prometheus graph for process_resident_memory_bytes.
3. Within two scrape intervals you should see RSS climb if vectors are in RAM, or stay flat if on_disk is set (then watch qdrant_filterable_ram_usage_bytes if you add filters).
4. Force a higher cap breach by lowering the alert threshold temporarily to 1e9 and re-run load. The QdrantMemoryCreep alert must appear in the Prometheus /alerts page.
5. Kill the loader. Memory should plateau or drop as the OS reclaims page cache. No OOM kill should appear in dmesg | grep oom (or kubectl logs for the node).

If the alert never fires or the metric stays frozen under load, your scrape target is wrong or the load is not reaching the node. Check qdrant_search_count to confirm queries arrive. This verification step is the difference between assuming monitoring works and knowing it does.

Step 7: Tune Qdrant to bound memory under query load

Once you can see the pressure, reduce it. Three levers are real:

• On-disk vectors: as shown, set "on_disk": True at collection creation. This moves the vector store out of RSS into mmap’d files.
• Limit concurrent searches: front Qdrant with a connection pool or use the batch search_batch API instead of hundreds of parallel single queries. Each in-flight search allocates a priority queue; bounding concurrency bounds that allocation.
• Trim payload indices: only index payload fields you filter on. Each indexed field adds to qdrant_filterable_ram_usage_bytes.

Example of a lean collection with on-disk vectors and a single payload index:

client.recreate_collection(
    collection_name="docs",
    vectors_config={"size": 768, "distance": "Cosine", "on_disk": True},
)
client.create_payload_index(
    collection_name="docs",
    field_name="tenant_id",
    field_schema="keyword",
)

After tuning, repeat Step 6. The derivative in deriv(process_resident_memory_bytes[5m]) should approach zero under the same load. That is the proof your Qdrant memory usage monitoring and tuning loop works.

What good looks like

A healthy high-load profile shows RSS flat or slowly oscillating within 60–70% of the limit, with qdrant_storage_ram_usage_bytes tracking the HNSW graph size and nothing else. When query rate doubles, the line stays put because vectors are mmap’d and concurrency is capped. The alert from Step 5 stays green until a real leak or misconfig appears.

Keep the dashboard up after you retire the temporary lowered threshold. Vector workloads drift as data grows, and the memory footprint of a collection with 10 million points is not the same as with 100 thousand. Revisit the scrape interval and alert threshold every time you change m or on_disk settings.