Setting up alerts for Pinecone index degradation

Pinecone index degradation alerts are rarely built into the dashboard by default, yet they are the difference between a silent retrieval failure and a paged on-call engineer. This tutorial walks through wiring up real-time monitoring of your index health, latency, and staleness using the Pinecone API, a small Python collector, and Prometheus.

Prerequisites

• A Pinecone project with an active index (serverless or pod-based).
• PINECONE_API_KEY with describe and query permissions.
• Python 3.10+ and pip install pinecone prometheus-client.
• A Prometheus instance (v2.45+) and Alertmanager.
• A Slack incoming webhook URL for notifications.
• Basic comfort with YAML and curl.

Why dashboard metrics aren’t enough

Pinecone’s console shows request volume and error rates at the project level. It does not tell you that a single index in a multi-tenant cluster is serving p99 latency of 800ms because of a skewed namespace. It also does not flag that your nightly ingestion job silently failed and vector count has been flat for six hours. Building your own Pinecone index degradation alerts closes that gap with per-index signals you control.

Step 1: Project setup

Create a virtual environment and install deps:

python -m venv venv
source venv/bin/activate
pip install pinecone prometheus-client

Export your key:

export PINECONE_API_KEY="pc-..."
export INDEX_NAME="prod-search"

Step 2: Build the metrics collector

The exporter polls describe_index_stats and runs a lightweight query to measure data-plane latency. We export four metrics: vector count, namespace count, query latency histogram, and error counter.

import os
import time
import random
from pinecone import Pinecone
from prometheus_client import start_http_server, Gauge, Histogram

PINECONE_API_KEY = os.environ["PINECONE_API_KEY"]
INDEX_NAME = os.environ.get("INDEX_NAME", "prod-search")
POLL_INTERVAL = int(os.environ.get("POLL_INTERVAL", "30"))

pc = Pinecone(api_key=PINECONE_API_KEY)
index = pc.Index(INDEX_NAME)

vector_count = Gauge("pinecone_index_vector_count", "Total vectors in index")
namespace_count = Gauge("pinecone_index_namespace_count", "Number of namespaces")
query_latency = Histogram("pinecone_query_latency_seconds", "Query latency")
error_count = Gauge("pinecone_index_errors_total", "Cumulative error count")
last_success = Gauge("pinecone_index_last_success_ts", "Timestamp of last successful poll")

def dummy_query():
    dim = index.describe_index_stats()["dimension"]
    vec = [random.random() for _ in range(dim)]
    start = time.time()
    index.query(vector=vec, top_k=1, include_values=False)
    return time.time() - start

def collect():
    try:
        stats = index.describe_index_stats()
        vector_count.set(stats["total_vector_count"])
        namespace_count.set(len(stats.get("namespaces", {})))
        lat = dummy_query()
        query_latency.observe(lat)
        last_success.set(time.time())
    except Exception as e:
        error_count.inc()
        print(f"poll failed: {e}")

if __name__ == "__main__":
    start_http_server(8000)
    while True:
        collect()
        time.sleep(POLL_INTERVAL)

Checkpoint: local verification

Run the script and scrape the endpoint:

curl -s localhost:8000/metrics | grep pinecone_index

Expected output (numbers vary):

pinecone_index_vector_count 124523.0
pinecone_index_namespace_count 3.0
pinecone_query_latency_seconds_count 4.0
pinecone_index_errors_total 0.0

If those lines appear, the collector is healthy.

Step 3: Scrape with Prometheus

Add a scrape job to prometheus.yml:

scrape_configs:
  - job_name: pinecone
    static_configs:
      - targets: ['localhost:8000']

Reload Prometheus (kill -HUP or --web.enable-lifecycle reload). In the expression browser, run up{job="pinecone"}. It returns 1. If it shows 0, check target health under /targets.

Step 4: Define alert rules

Create alerts.yml. The rules below fire on latency regression, unexpected vector loss, and repeated poll errors. These are the core Pinecone index degradation alerts we want paged.

groups:
  - name: pinecone_degradation
    rules:
      - alert: PineconeHighQueryLatency
        expr: histogram_quantile(0.99, sum(rate(pinecone_query_latency_seconds_bucket[5m])) by (le)) > 0.3
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Pinecone p99 query latency > 300ms"

      - alert: PineconeVectorCountDrop
        expr: changes(pinecone_index_vector_count[1h]) < 0 and abs(changes(pinecone_index_vector_count[1h])) > 1000
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Pinecone index lost vectors"

      - alert: PineconePollErrors
        expr: increase(pinecone_index_errors_total[10m]) > 3
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Pinecone exporter errors"

Launch Prometheus with --rules.file=alerts.yml. Validate syntax with promtool check rules alerts.yml.

Step 5: Route to Slack

Alertmanager config:

route:
  receiver: slack
  group_wait: 30s
receivers:
  - name: slack
    slack_configs:
      - api_url: ${SLACK_WEBHOOK}
        channel: '#alerts'
        text: "{{ .CommonAnnotations.summary }}: {{ .CommonAnnotations.description }}"

Restart Alertmanager. To test without waiting 10 minutes, temporarily lower the latency threshold to > 0.01 and add time.sleep(0.05) in dummy_query. After two scrape intervals, the alert appears in the Prometheus “Alerts” tab and posts to Slack:

PineconeHighQueryLatency: Index localhost:8000 p99 latency above baseline.

Step 6: Expand to multiple indexes

If you run several indexes, label the metrics and loop:

INDEX_NAMES = os.environ.get("INDEX_NAMES", "prod-search").split(",")
# inside collect, for name in INDEX_NAMES:
#   idx = pc.Index(name)
#   vector_count.labels(index=name).set(...)

Update alert expressions to include by (index, instance) where relevant.

Step 7: Calibrate thresholds

After a week of data, replace static values with baselines. For latency, use:

expr: histogram_quantile(0.99, sum(rate(pinecone_query_latency_seconds_bucket[1h])) by (le)) > (avg_over_time(pinecone_query_latency_seconds{quantile="0.99"}[1w]) * 1.5)

For ingestion staleness, predict_linear catches flatlines:

      - alert: PineconeStaleIngestion
        expr: predict_linear(pinecone_index_vector_count[1h], 3600) < pinecone_index_vector_count - 5000
        for: 30m

This detects silent write failures that raw error counters miss.

Operational notes

• Run the exporter as a Kubernetes sidecar or a tiny Lambda scheduled every minute; one query per 30s costs negligible CU.
• Use a read-only key scoped to describe/query.
• Never alert on up alone—Pinecone can be globally reachable while throttling your index.
• The dummy query should match your real top_k and filter shape to avoid optimistic latency.

Wrap-up

You now have a working pipeline that turns raw Pinecone stats into actionable Pinecone index degradation alerts. The pattern is portable: swap the client for any vector DB exposing a stats endpoint, and keep the Prometheus/Alertmanager half unchanged.