Multi-team API key governance at enterprise scale

At enterprise scale, handing every squad the same root credential is how you get a six-figure surprise bill and a SOC2 audit failure. Effective multi-team API key governance means issuing isolated, scoped, and observable keys per team, with enforced routing and rotation. This guide lays out an ordered path to get there without standing up a bespoke auth service that becomes a full-time job.

1. Kill the shared root key

The first step is mechanical but non-negotiable: the root key from your LLM provider lives only in a secrets manager, never in a .env that gets copied into ten repos. Shared keys make attribution impossible and revocation a denial-of-service event.

A common pitfall is thinking a single key with “different prefixes in logs” is governance. It isn’t. If the key is the same string, any team can impersonate another by changing a log field.

# Anti-pattern: same key everywhere
export OPENAI_API_KEY="sk-root-12345"  # in 40 repos

Pull it. Store it in Vault or AWS Secrets Manager with strict IAM. No human should paste it into Slack.

2. Model teams as namespaces

Before issuing keys, define a team identifier that maps to a cost center, not a person. Use a stable slug like payments-ml or doc-summarization. This slug becomes the primary dimension for metering and routing.

A minimal key metadata record looks like this:

{
  "team_id": "payments-ml",
  "scopes": ["model:gpt-4o", "model:claude-3-5-sonnet"],
  "max_tokens_per_day": 2000000,
  "expires_at": "2025-06-30T00:00:00Z"
}

Treat this as policy, not documentation. Your gateway or proxy must enforce every field. A tradeoff here is granularity: too many micro-teams creates key sprawl; too few defeats attribution. I land on one namespace per independently budgeted squad.

3. Issue scoped, signed keys

Don’t generate random strings and track them in a database if you can avoid it. Sign the metadata with HMAC using the root secret, and hand the team a token that contains the claims. The gateway verifies the signature and never needs a lookup.

import hmac, hashlib, base64, json, time

def issue_team_key(team_id, scopes, secret):
    payload = {
        "team_id": team_id,
        "scopes": scopes,
        "exp": int(time.time()) + 86400 * 30
    }
    body = base64.urlsafe_b64encode(json.dumps(payload).encode())
    sig = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
    return f"tk-{body.decode()}.{sig}"

# Usage
key = issue_team_key("payments-ml", ["model:gpt-4o"], "root-secret")

Tradeoff: lookup vs stateless

Stored keys let you revoke instantly but require a hot datastore on the request path. Signed keys are stateless but revocation needs a denylist or short TTL. I use signed keys with 30-day expiry and a small denylist for emergency revocation.

Pitfall: scope creep

Teams will ask for model:* because it’s easier. Say no. Broad scopes nullify your multi-team API key governance by letting one team spike another’s cost center.

4. Enforce routing and model access

A core piece of multi-team API key governance is preventing team A from burning quota on a frontier model they haven’t approved. Encode scopes in the key, then reject requests that violate them at the edge.

If you run a gateway that honors client routing directives, you can pass the team context in a header and let the gateway forward cache-control hints to the provider. For example, n4n.ai exposes a single OpenAI-compatible endpoint that respects x-team-routing and falls back automatically when a provider is degraded—handy when your policy says “prefer Anthropic, fall back to OpenAI”.

curl https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $TEAM_KEY" \
  -H "x-team-routing: anthropic:claude-3-5-sonnet;fallback:openai:gpt-4o" \
  -d '{"model":"claude-3-5-sonnet","messages":[{"role":"user","content":"hi"}]}'

Without that header, the gateway applies the team’s default policy. The point is that the team never sees provider keys; they see one endpoint and a routing rule.

Pitfall: silent fallback hiding cost

Automatic fallback is great for uptime but terrible for budgets if team A’s fallback model costs 10x. Log the resolved model on every response. Your audit stream should show route: openai even when the request asked for Anthropic.

5. Meter per token, not per request

Quota systems that count requests fail the moment someone batches 100k tokens. Your multi-team API key governance must hinge on usage.completion_tokens and usage.prompt_tokens from the provider response.

# Pseudo-code for metering middleware
async def track_usage(team_id, response):
    usage = response["usage"]
    total = usage["prompt_tokens"] + usage["completion_tokens"]
    await ledger.incr(f"team:{team_id}:tokens", total)
    if total > 50000:
        await alert(f"Heavy spend from {team_id}")

Per-token usage metering lets you charge back accurately and spot a runaway loop. If your gateway already returns this in the standard OpenAI shape, pipe it to your warehouse within the same job that writes the audit line.

Tradeoff: latency vs precision

Synchronous metering adds a write per call. For high-QPS teams, batch the increments every second. You lose real-time precision but keep the pipeline alive.

6. Automate rotation and revocation

Manual key rotation is a lie you tell yourself until the breach. Script it:

1. Issue new key for team.
2. Propagate to team’s secret store via API.
3. Wait one TTL for old key to drain.
4. Revoke old key.

# Rotate payments-ml key
NEW_KEY=$(python issue_key.py payments-ml)
vault kv put secret/payments-ml/api_key value="$NEW_KEY"
# after 24h
python revoke_key.py --old "$OLD_KEY"

Common pitfall: CI pipelines cache keys in images. If you rotate but the image isn’t rebuilt, the old key keeps working until the layer expires. Use external secret mounts, not baked values.

7. Audit with a flat schema

Your auditors want to know who called what, when, and at what cost. Emit one JSON line per inference:

{
  "ts": "2025-04-12T10:22:01Z",
  "team_id": "payments-ml",
  "model": "gpt-4o",
  "prompt_tokens": 1200,
  "completion_tokens": 300,
  "route": "openai",
  "key_id": "tk-ab12"
}

Ship to a partitioned table by team_id and ts. Don’t nest weirdly; flat is queryable. Retention should match your compliance window, not your disk budget.

8. Build vs buy, concretely

If you are comparing options, map the work:

• Self-hosted proxy (e.g., LiteLLM): You own the DB, the signing, the fallback logic. Good if you need air-gapped. Cost: you maintain it and absorb provider API changes.
• Provider-native sub-keys: Some providers offer project keys. They rarely span multiple model vendors, which breaks multi-team API key governance when team B wants Anthropic and team C wants Mistral. You end up with N key systems.
• OpenRouter-class gateway: One endpoint, 240+ models, automatic fallback, per-token metering. You still define team scopes, but the plumbing for routing and cache-control is done. The tradeoff is dependency on an external control plane unless you self-host the same open codebase.

The differentiator isn’t the key string; it’s whether the system can attribute, limit, and route by team without you writing a policy engine. If you already have a service mesh, slap a thin auth layer on a gateway. If you don’t, don’t underestimate the long tail of provider quirks—cache headers, streaming quirks, and rate-limit formats will eat a quarter.

9. Rollout checklist

Ordered path to production:

1. Vault the root key.
2. Define team slugs and cost centers.
3. Implement signed team keys (section 3).
4. Deploy gateway with scope enforcement.
5. Add per-token metering to warehouse.
6. Script rotation.
7. Enable audit logging.
8. Set alert thresholds per team.

Multi-team API key governance is not a feature you buy once. It’s a loop: issue, meter, rotate, audit. Do that and the audit question changes from “who spent this?” to “why was this team under budget?” The teams that treat keys as ephemeral credentials rather than permanent fixtures ship faster and sleep better.