Why you should never hardcode LLM API keys

A leaked LLM credential can incur thousands of dollars of unintended spend within hours, and recovering from a published key is pure cleanup drudgery. The discipline to never hardcode llm api keys is not bureaucratic hygiene—it is the difference between a contained incident and a compromised production account. Treat every key as a live financial instrument with full access to your model quota.

How keys actually escape

Most leaks are not sophisticated exploits. They are careless commits.

A developer copies a working snippet from a notebook into a React component. The string sk-... ships to every browser that loads the bundle. Scrapers scan public GitHub repos and NPM packages for that prefix continuously; the median time from commit to automated extraction is under five minutes.

Server-side leaks are quieter but just as fatal. Logging the request object for debugging, dumping environment on a health endpoint, or printing an exception that includes the client config will expose the secret in plaintext logs that flow to third-party aggregators.

When you never hardcode llm api keys, you force a boundary between code and credentials. That boundary is what lets you reason about blast radius.

Step 1: Move secrets to environment variables

The first actionable step is to remove every literal key from source and read it at runtime.

import os
from openai import OpenAI

# Raises KeyError if missing — fail fast, don't silently use a placeholder
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

For TypeScript services:

const apiKey = process.env.OPENAI_API_KEY;
if (!apiKey) throw new Error("OPENAI_API_KEY not set");

Pitfall: committing a .env file. Add .env* to .gitignore and use .env.example with empty values. Pre-commit hooks like gitleaks or trufflehog catch accidental adds.

Tradeoff: environment variables are visible to anyone with shell access on the host. They are a baseline, not a fortress.

Step 2: Upgrade to a secrets manager

For any production workload, environment variables should be injected from a dedicated secrets store, not written into a deploy config. AWS Secrets Manager, HashiCorp Vault, Google Secret Manager, or Doppler all solve the same problem: secrets live outside your repo and outside your image layers.

import boto3, os

def load_secret(name: str) -> str:
    client = boto3.client("secretsmanager")
    return client.get_secret_value(SecretId=name)["SecretString"]

# Pull at boot, cache in process memory
os.environ["OPENAI_API_KEY"] = load_secret("prod/llm/openai")

For Vault, the pattern is similar using hvac or a sidecar agent that writes to a tmpfs file.

Tradeoffs:

Added cold-start latency (one extra network call per instance).
IAM or policy complexity; misconfigured read permissions cause outages.
Secret versioning must be understood or you will pin a deleted version.

Teams that never hardcode llm api keys and instead centralize them in a manager can rotate credentials without rebuilding images.

Step 3: Never call providers directly from untrusted contexts

Browser code must never hold a provider key. The moment you ship it, it is public.

If your architecture requires client-side calls (e.g., a demo page), put a thin backend proxy in front:

import express from "express";
import { OpenAI } from "openai";

const app = express();
app.use(express.json());

const openai = new OpenAI({ api_key: process.env.OPENAI_API_KEY });

app.post("/v1/chat", async (req, res) => {
  // Validate shape, enforce rate limits, strip unknown fields
  const completion = await openai.chat.completions.create(req.body);
  res.json(completion);
});

app.listen(3000);

This proxy is also the right place to enforce tenant isolation, log token counts, and block prompt-injection patterns.

Routing through an OpenAI-compatible gateway such as n4n.ai collapses per-provider keys into a single org token; the gateway holds the real credentials, performs automatic fallback when a provider is degraded, and returns per-token usage metering without your services storing more than one secret. That reduces the number of places a leak can originate.

Step 4: Scope and rotate keys

A single global key reused across staging, prod, and CI is a liability. Create separate keys per environment and per service where your provider allows it.

Rotation procedure:

Generate a new key in the provider dashboard or via API.
Push the new value to the secrets manager under a new version.
Roll the service so it picks up the new value.
Disable the old key after confirming zero 401s in logs.

# Example: update Vault secret without generating a real key (placeholder)
vault kv put secret/llm/openai api_key="$NEW_KEY_FROM_PROVIDER"

Tradeoff: rotation adds operational steps. Automate it with a cron job or a CI task that calls the provider’s key endpoint, but never log the response.

If an employee leaves, rotation is non-negotiable. Long-lived keys outlive access reviews.

Step 5: Monitor and alert on anomalous usage

Metering is your early warning. Most providers expose per-key usage endpoints or dashboards; gateways often aggregate them.

If you use a gateway with per-token usage metering, ship those records to your metrics pipeline:

{
  "key_id": "org-proxy-token",
  "model": "gpt-4o-mini",
  "prompt_tokens": 1200,
  "completion_tokens": 340,
  "cost_usd": 0.0021
}

Set billing alerts at 25%, 50%, and 90% of expected monthly spend. A sudden spike at 2 a.m. from a key that should only serve EU traffic is a leak signal, not a feature.

Pitfall: only checking the dashboard weekly. By then the bill is real.

Common pitfalls and tradeoffs

Docker images

Baking ENV OPENAI_API_KEY=... into a Dockerfile is equivalent to hardcoding. Use build args only for non-secret config, and inject at runtime via orchestrator secrets.

Serverless consoles

AWS Lambda console shows environment variables in plaintext to anyone with lambda:GetFunction. Scope IAM tightly and prefer Secrets Manager references.

Shared CI keys

A key used in GitHub Actions that is also used in prod means a forked PR can exfiltrate it via a malicious step. Use separate keys with narrow permissions for CI.

Local development convenience

Developers paste keys into ~/.bashrc. That file syncs to backup services. Use a local secrets file with chmod 600 and never copy it to containers.

Ordered path to clean keys

Follow this sequence; do not skip steps because they feel basic.

Inventory every key in use. grep -r "sk-" . across repos, CI configs, and notebooks.
Extract all literals to environment variables. Delete from git history with git filter-repo if committed.
Migrate env values to a secrets manager and reference them at boot.
Proxy all untrusted client calls through a backend that holds the key.
Scope keys per environment and service; disable unused ones.
Rotate on a schedule and immediately on personnel changes.
Monitor token usage and billing with alert thresholds.

The cost of these steps is a few hours of plumbing. The cost of ignoring them is a forwarded Slack screenshot of your provider’s fraud notification.

Engineers who never hardcode llm api keys treat credential handling as a first-class component, not an afterthought. That posture scales from a weekend prototype to a multi-region product without a rewrite.