Most Qdrant deployments get monitored like a generic web service: CPU, memory, and a green check on the health endpoint. That coverage is useless when a vector search starts returning stale or partially missing results under load. Qdrant observability metrics need to focus on the signals that predict recall degradation, tail latency, and write stalls—not just whether the process is alive. This guide lays out an ordered, actionable path to instrument Qdrant for production LLM workloads.
1. Scrape the Prometheus endpoint from every node
Qdrant exposes a Prometheus endpoint at :6333/metrics (HTTP) or :6334/metrics (gRPC) on each node. In a cluster, you must scrape all peers, not just the leader. Raft replication lag and segment states are per-node, and a silent follower with a stuck optimizer will not show up if you only poll the coordinator.
scrape_configs:
- job_name: qdrant
metrics_path: /metrics
scrape_interval: 5s
static_configs:
- targets: ['qdrant-0:6333', 'qdrant-1:6333', 'qdrant-2:6333']If you run with TLS or API keys, configure authorization in the scrape job. Use metric relabeling to drop high-cardinality debug series such as per-collection internal timings unless you explicitly need them.
Pitfall: the default 15s scrape interval hides latency spikes that last 2–3 seconds during segment merges. Drop to 5s or lower for the search latency histogram. Tradeoff: tighter scraping increases Prometheus storage and cardinality. For a 10-node cluster at 5s, the qdrant_search_requests_duration_sec histogram with 12 buckets adds roughly 120 series per node—manageable, but don’t also scrape /debug/metrics or similar verbose endpoints.
2. Track p99 search latency, not averages
The average search time looks fine until your 99th-percentile query blocks a user-facing response. Qdrant emits qdrant_search_requests_duration_sec as a histogram for both REST and gRPC (labeled by transport). Use this PromQL for p99 across the cluster:
histogram_quantile(
0.99,
sum(rate(qdrant_search_requests_duration_sec_bucket[5m])) by (le)
)A common mistake is alerting on fleet-wide p99. A single hot collection will drown in aggregates. If your deployment serves multiple tenants, break it down by the collection label, or inject a tracing span and join with application logs.
Also watch qdrant_search_requests_total and the failure counter (typically qdrant_search_requests_failures_total). A rising failure rate with stable latency usually means timeouts from locked segments during heavy upserts.
Tradeoff: histogram buckets are fixed at export time. If your p99 sits between buckets (e.g., 150ms vs 200ms bucket), you get coarse resolution. Lower the bucket boundaries via Qdrant’s metrics config if you need finer insight—but adding more buckets multiplies series count.
3. Monitor the write path and segment count
Vector databases take writes incrementally and optimize in background. Qdrant creates new segments on upsert; an unoptimized segment pile-up destroys read performance. The metric qdrant_segments_count per collection is your leading indicator.
max(qdrant_segments_count{collection="docs"}) by (instance)If this climbs above ~20–30 without dropping, the optimizer is stalled—often due to disk I/O contention or CPU starvation from concurrent searches. Check qdrant_points_count for growth rate:
rate(qdrant_points_count{collection="docs"}[10m])Also track qdrant_upsert_requests_duration_sec to see if writes themselves are slowing. Pitfall: forcing optimization too aggressively with client.update_collection(optimizer_config=...) throttles reads. Let Qdrant’s default tuning work, but alert when segment count stays high for 10 minutes.
from qdrant_client import QdrantClient
client = QdrantClient(host="qdrant-0", port=6333)
# Trigger manual optimization only in emergencies, not on a schedule
client.update_collection(
collection_name="docs",
optimizer_config={"max_segment_size": 100000}
)Qdrant observability metrics for writes are incomplete without watching the WAL directory size on disk; a full disk halts upserts silently.
4. Measure recall with periodic offline probes
Qdrant observability metrics will not tell you if approximate search is silently missing neighbors. You must run recall probes. Pick a fixed set of 100 query vectors with known ground truth from a brute-force exact search, and compare to the HNSW results.
import random
from qdrant_client import models
# Assume `query_vec` is a representative production-style probe
exact = client.search(
collection_name="docs",
query_vector=query_vec,
limit=10,
search_params=models.SearchParams(exact=True)
)
approx = client.search(
collection_name="docs",
query_vector=query_vec,
limit=10,
search_params=models.SearchParams(hnsw_ef=128)
)
exact_ids = {p.id for p in exact}
recall = len(exact_ids & {p.id for p in approx}) / len(exact_ids)Run this every 15 minutes from a cron job against a shadow collection that mirrors production data. Plot recall@10 as a time series. If it drops below 0.95, increase hnsw_ef or rebuild the collection with a larger m parameter. Tradeoff: higher hnsw_ef improves recall but linearly increases search latency—watch the p99 from step 2.
Build your probe set from real queries sampled in production, not random vectors; random probes overestimate recall because they are evenly distributed.
5. Memory and disk saturation are the real killers
HNSW graphs live in RAM unless you use mmap storage. The process metric process_resident_memory_bytes is more important than CPU. A node that starts swapping will exhibit latency spikes that no Qdrant-specific metric captures.
process_resident_memory_bytes{job="qdrant"} / on(instance) group_left node_memory_total_bytesPitfall: container memory limits without cgroup awareness cause OOM kills during optimization. Set Qdrant’s max_memory in config to 70% of the container limit. Disk metrics matter for the write-ahead log; use node_filesystem_avail_bytes and alert at 15% free. If you use mmap, monitor page cache hit ratio via node_vmstat_pgmajfault—a rising major fault rate means vectors are being read from disk instead of RAM.
These Qdrant observability metrics for resources are the ones that prevent 3 a.m. pages about “slow search” that are actually memory pressure.
6. Separate embedding latency from search latency
In LLM pipelines, the query vector is produced by an embedding model. If you blame Qdrant for tail latency without measuring embedding time, you will misconfigure timeouts. When routing embeddings through an OpenRouter-class gateway such as n4n.ai, forward provider cache-control hints and track embedding p99 independently. Then subtract it from end-to-end latency to get true Qdrant observability metrics for search.
# Pseudocode for tracing
embed_start = time.time()
query_vec = embed(text, cache_key=text_hash)
embed_ms = (time.time() - embed_start) * 1000
search_start = time.time()
results = client.search(...)
search_ms = (time.time() - search_start) * 1000If embed_ms is 80% of total, tuning Qdrant is wasted effort. Export both as separate metrics so dashboards show the breakdown.
7. Alert rules that suppress noise
Finally, encode the above into alerts. Example Alertmanager rules:
groups:
- name: qdrant
rules:
- alert: QdrantP99High
expr: histogram_quantile(0.99, sum(rate(qdrant_search_requests_duration_sec_bucket[5m])) by (le)) > 0.2
for: 5m
- alert: QdrantSegmentsStuck
expr: max(qdrant_segments_count{collection="docs"}) by (instance) > 30
for: 10m
- alert: QdrantRecallDrop
expr: probe_recall_ratio{collection="docs"} < 0.95
for: 15m
- alert: QdrantMemorySaturation
expr: process_resident_memory_bytes{job="qdrant"} / node_memory_total_bytes > 0.85
for: 5mCommon pitfall: alerting on absolute memory bytes instead of saturation relative to limit. Use ratio. Another: not labeling alerts with collection name, so on-call can’t triage. Keep alert descriptions actionable: “Segment count stuck on qdrant-2, optimizer likely I/O blocked” beats “High segment count.”
Qdrant observability metrics only earn their keep when they map to user-visible failures. Follow this order—scrape, latency, writes, recall, resources, upstream, alerts—and you will catch the incidents that matter before they page a human.