How to store LLM API keys securely in production

A hardcoded OpenAI or Anthropic key in a Docker image is a ticking time bomb. To store LLM API keys securely production, you need a pipeline that keeps secrets out of source control, injects them at runtime via short-lived credentials, and rotates them on a schedule. The goal is not zero trust theater; it is reducing blast radius when—not if—a key escapes.

Step 1: Purge keys from source control

Before you build any vault, assume every key ever committed is burned. Search history, not just the working tree.

git log -p | grep -iE "sk-[a-zA-Z0-9]{20,}|api[_-]?key" || echo "no matches in diffs"

If you find a match, rotate that key at the provider immediately. Then add a pre-commit scanner so it cannot happen again:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.0
    hooks:
      - id: gitleaks

Run pre-commit install and fail CI if the scan errors. This is the floor for any plan to store LLM API keys securely production. Add a .gitignore entry for .env and any *.key files, but treat that as defense-in-depth, not primary control.

Step 2: Store secrets in a managed vault

Environment variables in a plain text .env file are not secret management. Use AWS Secrets Manager, GCP Secret Manager, or HashiCorp Vault. Below is a minimal AWS example.

aws secretsmanager create-secret \
  --name prod/llm/openai \
  --secret-string "$(cat openai_key.json)"

Retrieve it in Python without writing it to disk:

import boto3, json

def load_llm_key() -> str:
    client = boto3.client("secretsmanager")
    resp = client.get_secret_value(SecretId="prod/llm/openai")
    return json.loads(resp["SecretString"])["api_key"]

# Cache in memory with a short TTL to avoid throttling the secrets API
_cache = {}
def get_key_cached():
    if "key" not in _cache:
        _cache["key"] = load_llm_key()
        _cache["ts"] = time.time()
    return _cache["key"]

The vault handles encryption at rest with KMS; you handle encryption in transit by using the SDK over TLS. Never echo the secret back in a health-check response.

Step 3: Inject at runtime via workload identity

Do not ship the secret to the node. Let the runtime mint short-lived credentials. On EKS, use IAM Roles for Service Accounts (IRSA):

# serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: llm-worker
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/llm-secrets-reader

Your pod calls the Secrets Manager endpoint; AWS injects temp tokens via the metadata service. The static key never lands in a file or env var that ps or a core dump would expose. On GKE, the equivalent is Workload Identity:

metadata:
  annotations:
    iam.gke.io/gcp-service-account: llm-secrets@proj.iam.gserviceaccount.com

Same principle: the workload proves its identity to the secret store, and the store returns the cleartext only to the authorized pod.

Step 4: Apply least-privilege access policies

A secret reader role should only GetSecretValue on specific ARNs. Nothing more.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/llm/*"
    }
  ]
}

If a microservice only calls embeddings, it should not be able to read the chat-completions key. Compartmentalization limits blast radius when something does leak. Avoid wildcard * on Resource even in staging.

Step 5: Rotate on a schedule

Manual rotation is ignored under deadline pressure. Automate it. AWS supports rotation Lambdas; a minimal handler that calls the provider to issue a new key and updates the secret:

def lambda_handler(event, context):
    # provider-specific new key issuance omitted
    new_key = issue_new_provider_key()
    client = boto3.client("secretsmanager")
    client.put_secret_value(
        SecretId=event["SecretId"],
        SecretString=json.dumps({"api_key": new_key})
    )
    return {"status": "rotated"}

Set a 30- or 90-day rotation. Old keys should be revoked by the provider API in the same flow. This discipline is what lets you store LLM API keys securely production over the long term instead of hoping the leak never happens.

Step 6: Redact keys from logs and traces

The most common leak after source control is logging. Add a filter that masks anything resembling a key before it hits stdout.

import logging, re

class RedactFilter(logging.Filter):
    def filter(self, record):
        record.msg = re.sub(r"sk-[A-Za-z0-9]{20,}", "sk-***", str(record.msg))
        return True

logging.getLogger().addFilter(RedactFilter())

Verify your APM agent (Datadog, OTel) does not capture request bodies for the LLM client. Many SDKs echo the Authorization header in debug mode—turn that off. Add the filter at the root logger so third-party libraries inherit it.

Step 7: Consolidate provider keys behind a gateway

If you call OpenAI, Anthropic, and open-weights endpoints, you are managing three separate key lifecycles. Fronting them with a single OpenAI-compatible gateway collapses that to one secret. For example, n4n.ai exposes one endpoint covering 240+ models with automatic fallback when a provider is rate-limited, so you store a single gateway key server-side and route per request. The gateway honors client routing directives and forwards provider cache-control hints, so you keep control without handling raw provider credentials.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",
    api_key=get_key_cached()  # the one gateway secret from Step 2
)
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "ping"}],
    extra_headers={"x-n4n-route": "openai"}  # optional pinning
)

Your app code never sees the underlying provider keys. Rotation at the gateway level revokes access to all models at once.

Step 8: Verify the pipeline end to end

Run this checklist in a staging environment that mirrors production IAM:

1. git clone the repo fresh; confirm no key strings exist in tree or history.
2. Delete local .env; boot the service. It should start using the vault-injected credential.
3. Hit a /health endpoint that makes a real minimal LLM call (e.g., echo model). Expect 200.
4. Inspect container env: kubectl exec pod -- env | grep -i key returns nothing.
5. Tail logs during the call; confirm the redaction filter replaced any key material.
6. Trigger a manual rotation; confirm the service continues working without restart if using short-lived creds, or picks up new secret on rollout.

If all six pass, you have a repeatable way to store LLM API keys securely production that survives audits and incident reviews.

What success looks like

Success is boring: no secret in Git, no secret in env dump, one scoped IAM principal per service, and a rotation date in the calendar. When the next provider breach hits, you change one secret in one vault and move on.