Automating secret rotation with CI/CD pipelines

Most teams treat API keys as static credentials, but automating secret rotation ci/cd pipelines cuts blast radius when a token leaks. This guide walks through a concrete rotation workflow you can drop into GitHub Actions or GitLab CI, with code to generate, distribute, and revoke keys without service interruption.

Step 1: Define your rotation policy and secret backend

Pick a single source of truth for secrets. HashiCorp Vault, AWS Secrets Manager, or Google Secret Manager all support versioned secrets and fine-grained access policies. Your CI system should never store plaintext keys in repo files; it only fetches them at runtime using short-lived workload identity.

Set a rotation interval based on exposure risk. For high-traffic API keys, 30 days is conservative; 7 days is reasonable for internal services that call external LLM providers. Define a grace period (e.g., 24 hours) where both old and new keys are valid to avoid race conditions during rolling deploys.

Write the policy as code so it is reviewable:

{
  "rotation_interval_days": 7,
  "grace_period_hours": 24,
  "allowed_principals": ["arn:aws:iam::123456789012:role/ci-deploy"],
  "notify_topic": "arn:aws:sns:us-east-1:123456789012:sec-rotations"
}

If you use Vault, prefer dynamic secrets for databases, but for third-party API keys you still manage static credentials. Mount a KV v2 engine and enforce max_versions to prevent accumulation.

Step 2: Generate and stage a new secret in CI

Run a scheduled pipeline job that mints a new key and writes it as a new version in the secret store. The old version stays active under a PREVIOUS stage. Below is a Python script using boto3 to rotate a secret named prod/llm-api-key. It generates a high-entropy token and stages it.

import boto3, os, secrets, string, sys

def generate_key(length=32):
    alphabet = string.ascii_letters + string.digits
    return 'sk-' + ''.join(secrets.choice(alphabet) for _ in range(length))

def main():
    client = boto3.client('secretsmanager', region_name=os.environ['AWS_REGION'])
    new_key = generate_key()
    try:
        client.put_secret_value(
            SecretId='prod/llm-api-key',
            SecretString=new_key,
            VersionStages=['AWSCURRENT']
        )
        # Demote previous current to AWSPREVIOUS
        client.update_secret_version_stage(
            SecretId='prod/llm-api-key',
            VersionStage='AWSPREVIOUS',
            MoveToVersionId=client.get_secret_value(SecretId='prod/llm-api-key')['VersionId']
        )
        print("Staged new key version, old key retained as PREVIOUS")
    except Exception as e:
        print(f"Rotation failed: {e}", file=sys.stderr)
        sys.exit(1)

if __name__ == '__main__':
    main()

Wire this into a GitHub Actions workflow that triggers on a cron. Use OIDC instead of static AWS keys where possible:

name: rotate-secret
on:
  schedule:
    - cron: '0 3 * * 1'  # weekly Monday 3am UTC
permissions:
  id-token: write
  contents: read
jobs:
  rotate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - run: pip install boto3
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/ci-deploy
          aws-region: us-east-1
      - run: python rotate.py

Automating secret rotation ci/cd starts with this unattended generation step; never manually paste keys into a console.

Step 3: Distribute the new secret to runtime environments

After staging, the deploy job must pull the latest version and inject it into the target environment. For Kubernetes, patch the secret; for VMs, write to a file with 0600 perms. The deploy step is identical whether triggered by a feature merge or a rotation event.

Example kubectl patch in a CI step that runs after the rotation job succeeds:

NEW_KEY=$(aws secretsmanager get-secret-value --secret-id prod/llm-api-key \
  --version-stage AWSCURRENT --query SecretString --output text)
kubectl create secret generic llm-creds --dry-run=client \
  --from-literal=API_KEY="$NEW_KEY" -o yaml | kubectl apply -f -
kubectl rollout restart deployment/api-gateway
kubectl rollout status deployment/api-gateway --timeout=120s

If you use Vault, the equivalent is vault kv put secret/llm api_key=$NEW_KEY followed by a consul-template reload or a SIGHUP to the sidecar. The application must read from an env var or mounted file, never from a baked image layer.

For GitHub Actions deployments to serverless targets, export the key to the platform:

echo "API_KEY=$NEW_KEY" >> "$GITHUB_ENV"

Then in a later step, vercel env add or aws lambda update-function-configuration consumes it.

Step 4: Cut over traffic and verify health

Once the new secret is in place, the service picks it up on restart. Verify the new key works before revoking the old one. A minimal health check against your API:

curl -s -o /dev/null -w "%{http_code}\n" \
  -H "Authorization: Bearer $NEW_KEY" \
  https://api.example.com/v1/status

Expect 200. If you run multiple replicas, the load balancer spreads traffic; check a few individual pods via port-forward to be sure none cached the old key. Automating secret rotation ci/cd requires this check to be a blocking pipeline gate, not a manual ticket.

For an LLM inference gateway, validate the key against a cheap completion call. If you rotate keys for a service like n4n.ai, its per-token usage metering lets you confirm the old key stops accruing charges within minutes of revocation, which is a strong signal the cutover succeeded.

Step 5: Revoke the previous secret after grace period

Schedule a second job that runs after the grace window (e.g., 24 hours later). It disables or deletes the prior version. In AWS Secrets Manager, remove the AWSPREVIOUS stage and force-delete:

import boto3, os, sys

client = boto3.client('secretsmanager', region_name=os.environ['AWS_REGION'])
try:
    versions = client.list_secret_version_ids(SecretId='prod/llm-api-key')
    for v in versions['Versions']:
        if 'AWSCURRENT' not in v['VersionStages']:
            client.update_secret_version_stage(
                SecretId='prod/llm-api-key',
                VersionStage='AWSPREVIOUS',
                RemoveFromVersionId=v['VersionId']
            )
            client.delete_secret(
                SecretId='prod/llm-api-key',
                RecoveryWindowInDays=0,
                ForceDeleteWithoutRecovery=True
            )
            print(f"Revoked old version {v['VersionId']}")
except Exception as e:
    print(f"Revocation failed: {e}", file=sys.stderr)
    sys.exit(1)

For providers with explicit key management APIs, call them instead of just local deletion. Example using GitHub REST API to delete a PAT:

curl -X DELETE -H "Authorization: Bearer $ADMIN_TOKEN" \
  https://api.github.com/admin/tokens/$OLD_TOKEN_ID

Always emit an audit log entry with timestamps, job ID, and the secret version hashed (never log the plaintext).

Step 6: Verify success and audit

Verification is two-fold: confirm the new key is used everywhere, and confirm the old key is rejected. Run a negative test in the revocation pipeline:

HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer $OLD_KEY" \
  https://api.example.com/v1/status)
if [ "$HTTP_CODE" != "401" ]; then
  echo "ERROR: old key still valid"
  exit 1
fi

Expect 401. If any service still uses the old key, you’ll see 200 and must rollback the revocation by re-staging the old version.

Check CI logs for the rotation job’s exit code. Add a CloudWatch alarm on SecretRotationFailed and a Vault audit log scrape. For compliance, export the secret version history:

aws secretsmanager get-secret-value --secret-id prod/llm-api-key --version-stage AWSCURRENT

Automating secret rotation ci/cd closes the loop: the pipeline generates, deploys, and retires credentials on a fixed cadence. You reduce standing privilege and make breach containment a non-event.

Pitfalls to avoid

• Clock skew: If your grace period is shorter than max pod termination grace, some requests hit a dead key. Set terminationGracePeriodSeconds longer than your cutover sleep.
• Concurrent rotations: Use a distributed lock (DynamoDB lock or Vault lease) so two weekly jobs don’t overwrite each other.
• Local caching: Apps that cache credentials at startup must be forced to re-read. rollout restart is mandatory; configmap reload alone is not enough.
• Secret sprawl: Only store the active and previous versions. Enforce max_versions=2 in Vault KV v2 or prune with the script above.
• Missing permissions: The CI role needs secretsmanager:PutSecretValue and UpdateSecretVersionStage, but not DeleteSecret until the second job. Split privileges by job.

Following these steps gives you a repeatable, auditable rotation cycle that survives real incident response. The code samples are production-adjacent; adapt IAM roles, secret names, and provider endpoints to your environment.