Zero-downtime API key rotation for production apps

Running LLM inference in production means your credentials to providers will rotate—whether by policy, leak, or provider enforcement. Zero-downtime api key rotation production is not optional if you want to avoid 401 storms and cold restarts. Below is a battle-tested pattern to rotate keys live in a service that calls OpenAI-compatible endpoints, with runnable snippets and verification steps.

Step 1: Move keys out of static environment variables

If your API key is baked into a Docker image or passed as a fixed ENV at boot, you must redeploy to rotate. That is downtime by definition. Put the secret in a dynamic backend that supports read-at-runtime and versioning: HashiCorp Vault, AWS Secrets Manager, or GCP Secret Manager.

The app should fetch the key over the network on start and cache it for a short TTL. This keeps you decoupled from build pipelines.

import hvac
import time

class VaultKeySource:
    def __init__(self, addr, token, path, ttl=30):
        self.client = hvac.Client(url=addr, token=token)
        self.path = path
        self.ttl = ttl
        self._cache = None
        self._expiry = 0

    def get_key(self):
        if time.time() < self._expiry and self._cache:
            return self._cache
        resp = self.client.secrets.kv.v2.read_secret_version(path=self.path)
        self._cache = resp['data']['data']['api_key']
        self._expiry = time.time() + self.ttl
        return self._cache

Cache the value for 30 seconds. That bounds the propagation delay after you write a new version in Vault to half a minute—fast enough for incident response, slow enough to avoid hammering the backend.

Step 2: Implement a hot-reloadable key pool

Production services should support more than one valid key at a time. Providers like OpenAI allow you to issue multiple keys per org. A pool lets you add the new key before revoking the old one.

Build a thread-safe structure that refreshes from the source in the background.

import threading
import time

class KeyPool:
    def __init__(self, source, base_url):
        self.source = source
        self.base_url = base_url
        self.keys = {}  # key -> expiry epoch
        self.lock = threading.RLock()
        self._refresh()

    def _refresh(self):
        # source.fetch_all() returns [{"key": "...", "expires": 123456}]
        for entry in self.source.fetch_all():
            with self.lock:
                self.keys[entry['key']] = entry['expires']

    def start_background_refresh(self, interval=60):
        def loop():
            while True:
                time.sleep(interval)
                self._refresh()
        threading.Thread(target=loop, daemon=True).start()

    def active_keys(self):
        now = time.time()
        with self.lock:
            return [k for k, exp in self.keys.items() if exp > now]

Run start_background_refresh() once at process start. Your request path calls active_keys() and picks one (round-robin or random). No restart required when the underlying secret changes.

Step 3: Graceful cutoff for revoked keys

Never delete a key from the pool the instant you issue a replacement. Provider revocation can take seconds to minutes to propagate, and in-flight requests will fail if you pull the credential early.

When you rotate, write the new key to Vault with an expires timestamp set to now + grace_period (I use 300 seconds). The old key stays in the pool until its own expires passes. The refresh loop naturally drops it.

def rotate_via_vault(vault_client, path, new_key, grace=300):
    vault_client.secrets.kv.v2.patch(
        path=path,
        secret={"api_key": new_key, "expires": int(time.time() + grace)}
    )

This gives you zero-downtime api key rotation production behavior because the app always has at least one key that the provider still accepts.

Step 4: Atomic swap with a live health probe

Blindly trusting a new key is reckless. Probe it against the provider with a cheap call before advertising it to your request path.

import requests

def probe_key(key, base_url):
    r = requests.get(f"{base_url}/models",
                     headers={"Authorization": f"Bearer {key}"})
    return r.status_code == 200

def safe_add(self, new_key, expires, base_url):
    if not probe_key(new_key, base_url):
        raise RuntimeError("new key failed probe")
    with self.lock:
        self.keys[new_key] = expires

The probe should hit the same endpoint your app uses. For an OpenAI-compatible gateway, /models is a lightweight GET. If the probe fails, abort the rotation and alert—do not overwrite the working key.

Step 5: Decouple provider rotation from app via inference gateway

If your app talks to many model vendors, each with its own key lifecycle, the surface area multiplies. An inference gateway collapses that into a single credential on your side.

n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models and automatically falls back when a provider key is rate-limited or degraded. You rotate the provider credentials in its dashboard; your application continues sending the same gateway key. That removes the need for app-side hot reload entirely for provider-side rotations, though you still apply Steps 1–4 to the gateway key itself.

This is the cleanest way to achieve zero-downtime api key rotation production when you depend on heterogeneous backends.

Step 6: Verify success with canary traffic and metrics

Rotation is not done until you have evidence it worked. Stand up a canary loop that hammers your endpoint and counts auth errors.

#!/usr/bin/env bash
fail=0
for i in $(seq 1 600); do
  code=$(curl -s -o /dev/null -w "%{http_code}" https://app.internal/v1/chat \
    -H "Authorization: Bearer $APP_KEY" \
    -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"ping"}]}')
  if [ "$code" = "401" ]; then fail=$((fail+1)); fi
  sleep 1
done
echo "Total 401s during rotation window: $fail"

Expect 0. If you see any, your cutoff window was too short or the probe missed a permission scope.

In parallel, watch your metrics. A Prometheus query like sum(rate(http_requests_total{status="401"}[5m])) should stay flat. Alert if it spikes above baseline.

Step 7: Automate on schedule and on incident

Manual rotation breaks under pressure. Wire it to a cron or Vault lease renewal.

#!/usr/bin/env bash
# rotate.sh — runs weekly
NEW_KEY=$(openssl rand -hex 32)
vault kv put secret/llm api_key="$NEW_KEY" expires=$(($(date +%s)+300))
# trigger provider key creation out-of-band via provider CLI

For compromise response, call the same script from your incident runbook. Because the app picks up the change within its cache TTL, you contain a leak without a deploy.

Verification checklist

After any rotation, confirm:

1. active_keys() returns the new key and omits the old key after grace period.
2. Canary loop reports zero 401s across the swap window.
3. Provider dashboard shows the old key revoked (if you intended revocation).
4. Logs contain no key material—mask Authorization headers at the edge.

Zero-downtime api key rotation production is a matter of treating credentials as mutable runtime state, not compile-time constants. The code above is minimal but sufficient; harden the refresh loop with retries and circuit breakers before shipping to a hot path.