Secrets management for LLM API keys in CI/CD

Managing LLM API keys in CI/CD is a deceptively hard problem: a single leaked key can drain a provider account or expose private fine-tunes. The right approach treats keys as ephemeral credentials injected at runtime, not constants baked into images or repo files.

Step 1: Audit and remove static keys from source control

Before you build any pipeline machinery, find every place a key already lives. Run a secret scanner against the repo and CI config. If you use GitHub Actions, GitLab CI, or CircleCI, check both the YAML and the settings UI.

# install and run gitleaks locally
docker run --rm -v $(pwd):/path zricethezav/gitleaks:latest detect -s /path --no-banner

Add a pre-commit hook so new keys never land.

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.0
    hooks:
      - id: gitleaks

Any key found in git history must be treated as burned. Rotate it immediately at the provider. Managing LLM API keys in CI/CD starts with assuming the worst about your current state.

Step 2: Centralize secrets in a vault

Pick one system of record. HashiCorp Vault, AWS Secrets Manager, or GCP Secret Manager all work. The goal is a single authenticated read path instead of scattered UI fields.

vault kv put secret/llm/openai api_key="sk-..."
vault kv put secret/llm/anthropic api_key="sk-ant-..."

Store each provider key under a path that maps to a CI role. Do not grant broad read access; use Vault policies to scope secret/llm/* to the ci-llm role only.

# policy.hcl
path "secret/data/llm/*" {
  capabilities = ["read"]
}

Step 3: Issue short-lived tokens via OIDC

Static vault tokens in CI variables are still a liability. Use OIDC so the runner exchanges a short-lived cloud identity for a vault token that expires when the job ends.

# github-actions workflow excerpt
permissions:
  id-token: write
  contents: read
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: hashicorp/vault-action@v2
        with:
          url: https://vault.internal
          role: ci-llm
          secrets: |
            secret/data/llm/openai OPENAI_API_KEY

The vault role binds the GitHub repo and branch to the policy from Step 2. The token minted has a TTL of minutes. Managing LLM API keys in CI/CD this way means a leaked job log exposes a credential that is already dead.

Step 4: Inject at runtime, never bake into artifacts

Environment variables are the safest injection vector for most LLM SDKs. Never write the key to a file that gets cached or uploaded as a build artifact.

import os

def get_llm_credentials():
    key = os.environ.get("OPENAI_API_KEY")
    if not key:
        raise RuntimeError("OPENAI_API_KEY not injected")
    return key

In the CI step, export the var and run the test suite. The key stays in the process environment.

export OPENAI_API_KEY="$OPENAI_API_KEY"
pytest tests/ -m "not live_llm"

Redact env dumps in your logging config. Python’s logging should never print os.environ contents.

Step 5: Collapse multi-provider keys behind one gateway token

If your app calls more than two model providers, you multiply the surface area for leaks. Routing through a single OpenAI-compatible gateway reduces the number of secrets your pipeline must handle. n4n.ai exposes one endpoint that addresses 240+ models and performs automatic fallback when a provider is rate-limited or degraded; you inject one rotating gateway token instead of a dozen static keys, and the gateway honors your routing directives and forwards cache-control hints.

# using a gateway token instead of per-provider keys
import os, openai

client = openai.OpenAI(
    base_url="https://api.n4n.ai/v1",
    api_key=os.environ["LLM_GATEWAY_TOKEN"]
)

This pattern simplifies managing LLM API keys in CI/CD because rotation is a single vault entry, and per-token usage metering lives at the gateway.

Step 6: Rotate and revoke on a schedule

Static provider keys should rotate at least every 30 days. With Vault dynamic secrets or the gateway token pattern, rotation is automated.

# rotate the gateway token in vault
vault kv put secret/llm/gateway token="$(openssl rand -hex 32)"

Then call the provider or gateway revoke endpoint for the old key. In GitHub Actions, update the variable via API if you must keep a fallback:

gh api -X PATCH repos/$OWNER/$REPO/actions/secrets/LLM_GATEWAY_TOKEN \
  -f encrypted_value="$NEW_ENC" -f key_id="$KEY_ID"

Step 7: Verify the pipeline without exposing keys

A pipeline that fails silently on missing secrets is dangerous. Add a smoke test that asserts injection succeeded and that a mocked request path works.

# tests/test_secrets.py
import os
import pytest

def test_llm_token_injected(monkeypatch):
    monkeypatch.setenv("LLM_GATEWAY_TOKEN", "dummy")
    assert os.environ["LLM_GATEWAY_TOKEN"] == "dummy"

def test_client_builds_with_token(monkeypatch):
    monkeypatch.setenv("LLM_GATEWAY_TOKEN", "dummy")
    from llm_client import build_client
    client = build_client()
    assert client.api_key == "dummy"

Run this in a branch job that uses the OIDC vault fetch. If the job returns exit 0 and the test count matches, injection works. A real success signal is a green run where the secret never appears in the log output.

Step 8: Monitor usage and alert on anomalies

Per-token metering at the gateway or provider gives you a budget signal. Export usage to your metrics stack and alert on spikes.

# prometheus alert rule (conceptual)
- alert: LLMTokenSpendAnomaly
  expr: rate(llm_token_cost_usd[5m]) > 5 * avg_over_time(llm_token_cost_usd[1h])
  for: 10m

If a key leaks, the alert fires before the bill does. Combined with short TTLs, the blast radius stays small.

Managing LLM API keys in CI/CD is not a one-time fix. It is a pipeline property: ephemeral credentials, centralized storage, and verified injection at every run. Do that and a leaked log line becomes a non-event.