Manual key rotation through a ticketing system fails under scale. Automating API key rotation secrets manager integration gives you reproducible cycles, bounded blast radius, and an audit log that satisfies compliance without heroics. Below is an end-to-end pipeline using HashiCorp Vault and AWS, with patterns that port to GCP Secret Manager or Kubernetes.
Why programmatic rotation beats runbooks
A shell script someone runs locally is not rotation—it’s a liability with extra steps. The moment a key leaks, you need to know exactly which systems held it, what the new value is, and whether the old one is actually dead. A secrets manager makes that stateful and observable. The automation wraps the manager with provider APIs so the credential itself is never touched by human hands.
Step 1: Provision a secrets manager backend
Start with a real backend, not a JSON file in S3. For self-hosted, Vault is the default. Bring up a dev server to validate the pattern, then move to a sealed cluster with TLS.
vault server -dev -dev-root-token-id=root &
export VAULT_ADDR='http://127.0.0.1:8200'
vault login root
vault secrets enable -path=llm-keys kv-v2If you prefer managed, AWS Secrets Manager needs no server:
aws secretsmanager create-secret --name prod/llm/gateway \
--description "Rotated LLM provider key" --region us-east-1The path or ARN is your stable reference. Code reads from that, never from env vars baked into images.
Step 2: Define the credential schema and store the initial value
Treat the secret as a versioned document, not a bare string. Store metadata alongside the key so rotation logic can make decisions.
{
"api_key": "sk-old-1234567890",
"provider": "openai",
"created_at": "2024-05-01T12:00:00Z",
"rotates_at": "2024-05-15T12:00:00Z",
"status": "active"
}Write it with the Vault SDK so you get a version counter:
import hvac
client = hvac.Client(url="http://127.0.0.1:8200", token="root")
client.secrets.kv.v2.create_or_update_secret(
path="gateway",
secret={
"api_key": "sk-old-1234567890",
"provider": "openai",
"created_at": "2024-05-01T12:00:00Z",
"rotates_at": "2024-05-15T12:00:00Z",
"status": "active",
},
mount_point="llm-keys",
)Step 3: Implement the rotation function
The core job: call the provider to issue a new key, confirm it works, then write it as a new version. Using AWS IAM as a stand-in for any provider with a create-key API:
import boto3
import datetime
import hvac
def rotate_iam_key(user_name: str, vault_path: str):
iam = boto3.client("iam")
# create new access key
resp = iam.create_access_key(UserName=user_name)
new_key = resp["AccessKey"]["AccessKeyId"]
new_secret = resp["AccessKey"]["SecretAccessKey"]
# minimal live test
sts = boto3.client(
"sts",
aws_access_key_id=new_key,
aws_secret_access_key=new_secret,
)
sts.get_caller_identity()
client = hvac.Client(url="http://127.0.0.1:8200", token="root")
client.secrets.kv.v2.create_or_update_secret(
path=vault_path,
secret={
"api_key": new_key,
"api_secret": new_secret,
"provider": "aws-iam",
"created_at": datetime.datetime.utcnow().isoformat() + "Z",
"rotates_at": (datetime.datetime.utcnow() + datetime.timedelta(days=30)).isoformat() + "Z",
"status": "active",
},
mount_point="llm-keys",
)
# disable old key after grace period in a separate step
return new_keyFor LLM providers with a REST endpoint, replace the boto3 call with requests.post to their key-management route. The pattern holds: issue, verify, write new version, schedule old-key revocation.
Step 4: Wire the automated trigger
Don’t rely on a cron on someone’s laptop. Use a managed scheduler. On AWS, EventBridge + Lambda is clean:
aws events put-rule --schedule-expression "rate(30 days)" --name llm-key-rotation
aws lambda create-function --function-name rotate-llm-key \
--runtime python3.12 --handler rotate.lambda_handler \
--role arn:aws:iam::123456789012:role/rotator --zip-file fileb://deploy.zip
aws events put-targets --rule llm-key-rotation --targets "Id=1,Arn=arn:aws:lambda:us-east-1:123456789012:function:rotate-llm-key"Locally or in Vault, a systemd timer or Kubernetes CronJob works equally well. The job must be idempotent: if it runs twice, it should not create two live keys without revoking the prior.
Step 5: Propagate the new key to running services
A rotated secret is useless if your app cached the old one. Two patterns: push or pull. Pull is simpler with Vault Agent or External Secrets Operator.
Vault Agent config injects the latest version into a file:
template {
destination = "/etc/secrets/llm_key"
content = "{{ with secret \"llm-keys/data/gateway\" }}{{ .Data.data.api_key }}{{ end }}"
}For Kubernetes, External Secrets maps the Vault path to a Secret the deployment mounts:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: llm-gateway-key
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: llm-gateway-key
data:
- secretKey: api_key
remoteRef:
key: llm-keys/data/gateway
property: api_keyYour service must re-read the file or env on SIGHUP. A small TS snippet for an Express app:
import fs from 'fs';
let apiKey = fs.readFileSync('/etc/secrets/llm_key', 'utf8');
process.on('SIGHUP', () => {
apiKey = fs.readFileSync('/etc/secrets/llm_key', 'utf8');
});Step 6: Schedule old-key revocation with a grace window
Immediate revocation risks dropping in-flight requests. Write the old key to a deprecated status, keep it for 24h, then delete.
def revoke_old_key(iam_user: str, old_key_id: str):
iam = boto3.client("iam")
iam.update_access_key(
UserName=iam_user, AccessKeyId=old_key_id, Status="Inactive"
)
# after grace period
iam.delete_access_key(UserName=iam_user, AccessKeyId=old_key_id)Automate the second call with a delayed queue message or a separate CronJob.
Step 7: Verify success
Verification is not “the script exited 0”. Confirm the new key authenticates and the old one fails.
import requests
def verify(vault_path: str, endpoint: str):
client = hvac.Client(url="http://127.0.0.1:8200", token="root")
secret = client.secrets.kv.v2.read_secret_version(path=vault_path, mount_point="llm-keys")
key = secret["data"]["data"]["api_key"]
r = requests.get(endpoint, headers={"Authorization": f"Bearer {key}"})
assert r.status_code == 200, "new key rejected"
# old key should be inactive
print("rotation verified, version", secret["data"]["metadata"]["version"])Run this as a post-rotation check in CI or as a Lambda step. If it fails, roll back by marking the previous version active.
Cutting the rotation surface with a gateway
If you front multiple model providers through a single OpenAI-compatible endpoint, you rotate one gateway credential instead of ten per-vendor keys. n4n.ai addresses 240+ models behind one endpoint and honors client routing directives, so the secrets-manager job only needs to cycle that single token while provider-specific keys stay internal to the gateway. That shrinks the automation scope and the audit matrix.
Common pitfalls
Race conditions happen when two rotators run concurrently. Use a distributed lock—Vault’s sys/leases or DynamoDB conditional writes—before issuing new keys.
Downtime comes from apps that read env at boot and never reload. Fix the app, don’t weaken rotation frequency.
Missing rollback turns a leak into an outage. Always keep the prior version retrievable for at least one cycle.
Final checklist
- Secrets manager backed by policy and audit log.
- Rotation function issues, verifies, writes new version.
- Scheduler triggers idempotently.
- Workloads pull updated secret with reload signal.
- Old key revoked after grace window.
- Verification step asserts live auth.
Automating API key rotation secrets manager pipelines is infrastructure, not scripting. Build it like you would any other critical service.