Revoking compromised API keys without breaking production

Revoking compromised api keys production environments is a high-stakes operation that demands both speed and precision. You can do it without downtime if you treat key rotation as a controlled shift of trust rather than a hard cutover. This guide walks through the exact steps we use to contain a leak and rotate credentials while traffic keeps flowing.

Step 1: Confirm the compromise and scope blast radius

Before touching anything, verify the key is actually exposed and map what it protects. Check access logs for anomalous IPs, sudden quota spikes, or calls from regions your service never hits. A leaked key used by an attacker will show up as a distinct signature against your normal traffic shape.

# Example: scan Nginx JSON logs for a specific key fingerprint
grep '"api_key":"sk-old456"' /var/log/nginx/access.log \
  | jq -r '.remote_addr' | sort | uniq -c | sort -rn | head

If you cannot confirm misuse but suspect the key leaked (e.g., committed to a public repo), assume worst case. The cost of a false negative is far higher than a rotation exercise.

Step 2: Generate and store a replacement key

Create a new key at the provider using their admin API or console. Never paste it directly into source code or CI variables that get baked into images. Push it to your secret manager alongside the existing key so both coexist.

# AWS Secrets Manager: store a JSON list of active keys
aws secretsmanager put-secret-value \
  --secret-id prod/llm-keys \
  --secret-string '{"keys":["sk-new9abc","sk-old456"]}'

The replacement key should have identical scoped permissions. If your provider supports per-key rate limits or project isolation, mirror the old configuration exactly to avoid surprising 403s after cutover.

Step 3: Deploy the new key alongside the old (dual-key)

Your application must be able to read multiple keys and prefer the new one while keeping the old as fallback. This is the core of revoking compromised api keys production systems without breaking them: you run both until you are certain the new path is healthy.

A minimal loader that pulls from Secrets Manager at startup:

import boto3, json, os

def load_key_ring(secret_id="prod/llm-keys"):
    client = boto3.client("secretsmanager")
    resp = client.get_secret_value(SecretId=secret_id)
    return json.loads(resp["SecretString"])["keys"]

class KeyRing:
    def __init__(self, keys):
        # index 0 = primary, rest = fallbacks
        self.keys = keys

    def primary(self):
        return self.keys[0]

    def all(self):
        return self.keys

If you front LLM calls with a gateway such as n4n.ai, you can register the new upstream key at the gateway and switch routing without touching application code; the gateway’s automatic fallback also masks transient provider errors during the cutover.

Client-side retry with key fallback

Wrap your HTTP calls so a 401 on the primary key automatically retries with the next key. This protects you during the window where some nodes still hold only the old key.

import requests

def call_llm(payload, keyring):
    last_err = None
    for key in keyring.all():
        try:
            r = requests.post(
                "https://api.openai.com/v1/chat/completions",
                headers={"Authorization": f"Bearer {key}"},
                json=payload,
                timeout=30,
            )
            if r.status_code == 200:
                return r.json()
            if r.status_code == 401:
                last_err = "auth"
                continue
            r.raise_for_status()
        except requests.RequestException as e:
            last_err = e
    raise RuntimeError(f"all keys failed: {last_err}")

Step 4: Shift traffic to the new key gradually

Flip the primary index in your secret from sk-old456 to sk-new9abc and let the secret propagate. If you use a long-lived cache, force a refresh or roll your instances. For stateful services, a canary is safest: route 10% of pods to the new key only, watch error rates, then expand.

# Promote new key to primary, keep old as fallback
aws secretsmanager put-secret-value \
  --secret-id prod/llm-keys \
  --secret-string '{"keys":["sk-new9abc","sk-old456"]}'

Your KeyRing.primary() now returns the new key. The old key remains in the list as a safety net for any straggler process that hasn’t picked up the update.

Step 5: Revoke the compromised key at the provider

Once telemetry shows zero 401s on the old key for a full traffic cycle (we use 30 minutes minimum), revoke it at the issuer. Use the provider’s documented endpoint—do not rely on deletion alone.

For OpenAI organization keys, the admin call is:

curl -X DELETE "https://api.openai.com/v1/organization/keys/key-old456" \
  -H "Authorization: Bearer $OPENAI_ADMIN_KEY"

For AWS IAM, deactivate then delete the access key:

aws iam deactivate-access-key --access-key-id AKIAOLD456 --user-name svc-llm
aws iam delete-access-key --access-key-id AKIAOLD456 --user-name svc-llm

This is the point of no return. If your dual-key deployment was correct, no live request uses it.

Step 6: Verify zero 401s and monitor

Verification is not optional. Run a synthetic request with the revoked key from a isolated test context; it must return 401.

import requests

def assert_revoked(old_key):
    r = requests.get(
        "https://api.openai.com/v1/models",
        headers={"Authorization": f"Bearer {old_key}"},
        timeout=10,
    )
    assert r.status_code == 401, f"old key still valid: {r.status_code}"
    print("revocation confirmed")

assert_revoked("sk-old456")

In production, watch your gateway or application metrics for 401 spikes. A sudden rise means a cached client or a missed deployment still references the dead key. The hardest part of revoking compromised api keys production teams share is catching those late-binding workers.

What success looks like

• Old key returns 401 from provider.
• Production error rate unchanged before and after revocation.
• Secret store contains only the new key.

Step 7: Clean up and document

Remove the old key from the secret entirely so it cannot be accidentally re-enabled.

aws secretsmanager put-secret-value \
  --secret-id prod/llm-keys \
  --secret-string '{"keys":["sk-new9abc"]}'

Write a short incident note: time of detection, exposure vector, rotation duration, and any near-misses. If the leak came from a repo or log, fix the source before declaring done. After revoking compromised api keys production postmortems should feed back into tighter scopes and shorter key TTLs.

Key takeaways for engineers

• Always design clients to accept multiple keys; single-key assumptions turn rotation into an outage.
• Store keys in a manager that supports atomic multi-value updates.
• Revoke only after the old key is demonstrably idle in live traffic.
• Verify with a negative test (expect 401) rather than assuming silence means safety.

Treat every key as burnable. The moment you can rotate without a deployment freeze, your incident response gets exponentially cheaper.