Monitoring canary metrics during a model version rollout

Shipping a new model version to 100% of users without evidence is reckless. Monitoring canary metrics for model rollout lets you route a small traffic slice to the candidate, compare its behavior against the incumbent, and roll back before users notice regressions. This post gives you an end-to-end workflow with code you can adapt.

Step 1: Define the canary split with a routing flag

Start by deciding how much traffic the candidate model receives. Use a feature flag or a deterministic hash so the same user consistently hits the same model version.

# routing.py
CANARY_WEIGHT = 0.05  # 5% of traffic
BASELINE_MODEL = "gpt-4o-2024-05-13"
CANARY_MODEL = "gpt-4o-2024-08-01-preview"

def select_model(user_id: str) -> str:
    # stable assignment: same user always gets same bucket
    bucket = (hash(user_id) % 100) / 100.0
    return CANARY_MODEL if bucket < CANARY_WEIGHT else BASELINE_MODEL

If you front your inference with n4n.ai, its OpenAI-compatible endpoint addresses 240+ models and honors client routing directives, so you can pin the canary slice by sending a model alias per request while automatic fallback covers provider degradation.

Wire this into your request path before calling the model:

model = select_model(user_id)
resp = client.chat.completions.create(model=model, messages=...)

Step 2: Instrument the request path with structured metrics

You cannot manage what you do not measure. Emit per-request metrics tagged with model version and canary status. Use Prometheus or OpenTelemetry.

# metrics.py
from prometheus_client import Histogram, Counter, start_http_server
import time

REQUEST_LATENCY = Histogram(
    'llm_request_latency_seconds',
    'End-to-end LLM call latency',
    ['model', 'canary']
)
TOKEN_COUNTER = Counter(
    'llm_tokens_total',
    'Token usage',
    ['model', 'canary', 'type']
)
ERROR_COUNTER = Counter(
    'llm_errors_total',
    'Failed requests',
    ['model', 'canary', 'error_type']
)

def tracked_completion(client, model, canary, **kwargs):
    start = time.time()
    try:
        resp = client.chat.completions.create(model=model, **kwargs)
        REQUEST_LATENCY.labels(model, canary).observe(time.time() - start)
        TOKEN_COUNTER.labels(model, canary, 'prompt').inc(resp.usage.prompt_tokens)
        TOKEN_COUNTER.labels(model, canary, 'completion').inc(resp.usage.completion_tokens)
        return resp
    except Exception as e:
        ERROR_COUNTER.labels(model, canary, type(e).__name__).inc()
        raise

Start the exporter in your service:

start_http_server(8000)

Step 3: Emit canary-specific labels consistently

The canary label must be a string "true" or "false", not a boolean, to keep Prometheus happy. Derive it from your routing decision:

canary_label = "true" if model == CANARY_MODEL else "false"
tracked_completion(client, model, canary_label, messages=messages)

Log the same label in your structured logs. You will need it later to join online metrics with offline eval results.

{
  "ts": "2024-09-12T10:22:01Z",
  "user_id": "u_123",
  "model": "gpt-4o-2024-08-01-preview",
  "canary": "true",
  "prompt_tokens": 120,
  "completion_tokens": 45,
  "latency_ms": 820
}

Step 4: Build a dashboard with comparative panels

Monitoring canary metrics for model rollout requires side-by-side views. In Grafana, create panels that overlay canary and baseline series.

Latency p95:

histogram_quantile(
  0.95,
  sum(rate(llm_request_latency_seconds_bucket{canary="true"}[5m])) by (le, model)
)

Baseline p95 for contrast:

histogram_quantile(
  0.95,
  sum(rate(llm_request_latency_seconds_bucket{canary="false"}[5m])) by (le, model)
)

Token throughput:

sum(rate(llm_tokens_total{canary="true", type="completion"}[5m])) by (model)

Add a ratio panel: canary latency divided by baseline latency. A ratio above 1.2 for ten minutes is a smell.

Step 5: Set up alerting on divergence

Dashboards are for humans; alerts are for sleep. Define thresholds that compare canary to baseline.

# prometheus/rules.yml
groups:
  - name: canary
    rules:
      - alert: CanaryLatencySpike
        expr: |
          histogram_quantile(0.95, sum(rate(llm_request_latency_seconds_bucket{canary="true"}[5m])) by (le))
          >
          1.5 * histogram_quantile(0.95, sum(rate(llm_request_latency_seconds_bucket{canary="false"}[5m])) by (le))
        for: 10m
        labels:
          severity: page
      - alert: CanaryErrorRate
        expr: |
          sum(rate(llm_errors_total{canary="true"}[5m]))
          /
          sum(rate(llm_request_latency_seconds_count{canary="true"}[5m]))
          > 0.02
        for: 5m

Route these to Slack or PagerDuty. The goal is to catch a bad rollout inside the canary window.

Step 6: Run offline evals on a canary sample

Online metrics miss silent quality regressions. Capture a sample of canary inputs and outputs, then score them with your eval harness.

# eval_canary.py
import json
from my_eval_lib import score_response

THRESHOLD = 0.7
breaches = 0
total = 0

with open('canary_logs.jsonl') as f:
    for line in f:
        rec = json.loads(line)
        if rec.get('canary') != 'true':
            continue
        total += 1
        eval_score = score_response(rec['prompt'], rec['response'])
        if eval_score < THRESHOLD:
            breaches += 1

if total > 0 and breaches / total > 0.1:
    print("Canary eval failure rate too high")
    # trigger rollback (see Step 7)

Keep the eval set fixed across rollouts so numbers are comparable.

Step 7: Automate rollback on metric breach

Manual rollback is too slow at 3 a.m. Call your flag service API when an alert fires.

# rollback.py
import requests

FLAG_API = "https://flags.internal/api/v1/flag/canary-model"
API_TOKEN = "xxx"  # from secret store

def disable_canary():
    r = requests.post(
        f"{FLAG_API}/disable",
        headers={"Authorization": f"Bearer {API_TOKEN}"},
        timeout=5,
    )
    r.raise_for_status()

if __name__ == "__main__":
    disable_canary()

Wire this to a Prometheus alert webhook or a cron that checks the eval output.

Verify success

You know the pipeline works when:

1. A synthetic request with a hashed user ID in the canary bucket hits the candidate model (check logs for canary:"true").
2. The metrics endpoint exposes llm_request_latency_seconds with both canary="true" and canary="false" series.
3. The Grafana panel shows two lines after traffic flows for five minutes.
4. Triggering disable_canary() flips the flag and new requests route to baseline.
5. Inject an artificial latency in the canary path and confirm the alert fires within the for: window.

Monitoring canary metrics for model rollout is not a one-time task. Revisit thresholds as traffic shape changes, and keep the eval set honest. The moment you stop measuring, the canary stops protecting you.