API key rotation: a practical policy for LLM platforms

Most LLM platforms treat API keys as static credentials, which is a mistake that expands blast radius when a leak occurs. An api key rotation policy llm platforms can enforce must cover two distinct classes: the secrets you hold for upstream model providers and the keys you issue to your own tenants. Without a defined lifecycle, you will eventually face a midnight revocation that breaks production traffic and burns engineering time.

Identify your key classes

Before writing any policy, separate the keys by trust boundary. Upstream provider keys authenticate your platform to OpenAI, Anthropic, or a gateway that aggregates models behind one endpoint. Tenant keys authenticate external developers calling your API. Internal service keys link microservices within your own infra.

Each class has different rotation tolerance. Provider keys often have no native “rotate without downtime” API; you must create a second key and swap configuration. Tenant keys can be designed by you to support multiple concurrent active keys. Internal keys should follow the same pattern as tenant keys but with stricter network policies.

Ignore this distinction and you will either over-rotate provider keys causing outages, or under-rotate tenant keys increasing risk. Map every key to one of these classes in your secret inventory.

Define rotation triggers

A practical policy uses both time-based and event-based triggers.

Time-based: rotate provider keys every 90 days, tenant keys every 30–60 days depending on contract, internal keys every 30 days. Event-based: any suspected exposure, employee offboarding, or permission scope change forces immediate rotation.

Tradeoff: aggressive rotation shrinks attack window but multiplies operational toil. If you rotate tenant keys daily, your customers will hardcode them and bypass your rotation flow. Set the interval based on the cost of a leak versus the cost of a broken integration. Compliance frameworks like SOC 2 explicitly expect documented rotation periods; treat the policy as a control.

Dual-key overlap for zero downtime

Never delete the old key before the new one is proven live. For upstream keys, script the creation of a new secret, push it to your config store, and reload the client. Keep the old key active for a 24-hour grace window.

Example using HashiCorp Vault for a provider key:

# Write new version of provider key
vault kv put secret/llm/provider/openai api_key="sk-new123"
# Old version remains readable until we delete it explicitly

In your gateway config, reference the Vault path, not the literal key. On reload, the client picks up the new value. A minimal config snippet might look like:

secret_config:
  vault_path: "secret/llm/provider/openai"

For tenant keys, issue a versioned key pair:

{
  "tenant_id": "acme",
  "active_keys": [
    {"id": "k1", "secret": "tk_old", "expires": "2024-08-01T00:00:00Z"},
    {"id": "k2", "secret": "tk_new", "expires": "2024-09-01T00:00:00Z"}
  ]
}

Clients can present either key; your auth layer validates against the list. After the old key expires, drop it from the list. This overlap removes the classic “we rotated and now 5% of clients fail” incident.

Automate issuance and revocation

Manual rotation does not scale past a handful of keys. Use a secrets manager with short TTLs and automated renewal. For tenant self-service, expose a rotate endpoint that returns a new key and schedules the old one for expiry.

Python snippet using boto3 to rotate an AWS Secrets Manager secret:

import boto3, secrets, string

def rotate_tenant_key(secret_id):
    client = boto3.client('secretsmanager')
    new_key = 'tk_' + ''.join(secrets.choice(string.ascii_letters+string.digits) for _ in range(32))
    client.put_secret_value(SecretId=secret_id, SecretString=new_key,
                            VersionStages=['AWSCURRENT'])
    # Previous version automatically moves to AWSPREVIOUS
    return new_key

The staging labels let your auth layer accept both AWSCURRENT and AWSPREVIOUS for a window. A Flask route for tenant rotation:

from flask import Flask, request
app = Flask(__name__)

@app.route('/v1/tenants/<tid>/rotate', methods=['POST'])
def rotate(tid):
    secret_id = f"tenant/{tid}/key"
    new_key = rotate_tenant_key(secret_id)
    return {"new_key": new_key, "old_key_valid_for_hours": 24}

Pitfall: if your code caches keys in memory for hours, a rotation event won’t propagate. Use a short cache TTL (under 60s) or a pub/sub invalidation channel.

Handle in-flight requests

Rotation must not drop requests that are mid-flight. For upstream provider calls, use a retry policy that reads the latest key from your config on 401. For tenant traffic, the dual-key overlap covers it, but clients with cached keys need a clear error suggesting re-fetch.

If you run a gateway that honors client routing directives, ensure it forwards the correct key version to the upstream. A gateway such as n4n.ai, which provides per-token usage metering and honors client routing directives, lets you attribute traffic to each key version during the overlap window. This makes it trivial to confirm the old key is no longer sending tokens before you pull it.

Connection pools complicate this: an HTTP client with keep-alive may hold a connection authenticated under the old key. Close pools on key change or use per-request auth headers.

Audit and verify

After rotation, verify the old key is dead. Hit the endpoint with the revoked key and expect 401. Check logs for any successful auth with the old key after expiry—that indicates a stale client or cache bug.

Per-token metering helps here: you can query usage grouped by key ID for the last 48 hours. If the old key shows zero tokens post-expiry, revocation succeeded. Without metering, you are guessing based on sparse access logs.

Common pitfalls

• Hardcoding in CI: keys committed to repos surface in git history. Use ephemeral CI secrets and scan PRs with trufflehog.
• No rollback: if the new key is mis-copied, you need to revert fast. Keep the old key active until health checks pass.
• Provider rate limits during bulk rotation: rotating 100 tenant keys at once may trigger your own internal API limits. Stagger with a job queue.
• Assuming providers support instant revoke: some providers take minutes to propagate revocation. Design for lag; keep overlap windows longer than their documented SLA.
• Ignoring cache-control: if you forward provider cache-control hints, a cached response may outlive the key that fetched it. Treat cached payloads as independent of key validity.

Policy template as code

Store the policy in your repo as a JSON document reviewed quarterly:

{
  "policy": "api key rotation policy llm platforms",
  "classes": {
    "provider": {"max_age_days": 90, "overlap_hours": 24},
    "tenant": {"max_age_days": 45, "overlap_hours": 24},
    "internal": {"max_age_days": 30, "overlap_hours": 12}
  },
  "event_triggers": ["leak", "offboard", "scope_change"],
  "automation": "vault+boto3",
  "audit": "per_token_metering"
}

A minimal actionable api key rotation policy llm platforms should adopt this structure. The goal is not perfect security but bounded blast radius and predictable operations. Rotate like you mean it, but never at 2am without overlap.