AWS Secrets Manager vs HashiCorp Vault for LLM API keys

Choosing where to store LLM provider credentials forces a pragmatic decision: aws secrets manager vs vault api keys is a debate that pits fully managed convenience against self-hosted control. For teams shipping LLM features, the trade-offs hit rotation speed, latency at inference time, and audit depth.

Capabilities

Both systems store arbitrary key-value blobs, but their feature sets diverge quickly once you move past “put” and “get”. AWS Secrets Manager provides managed rotation via Lambda, versioned secrets with staging labels (AWSCURRENT, AWSPREVIOUS), and tight IAM integration. You store a key, attach a rotation schedule, and AWS invokes a Lambda that calls the provider’s key endpoint. Vault offers static secret versioning through the KV v2 engine, plus dynamic secrets, PKI, transit encryption, and a broad set of auth methods. For LLM API keys, dynamic secrets are irrelevant—providers like OpenAI, Anthropic, and Mistral issue static keys—so the differentiator is rotation ergonomics and audit granularity.

Fetching from AWS SM in Python:

import boto3
sm = boto3.client('secretsmanager', region_name='us-east-1')
resp = sm.get_secret_value(SecretId='openai-prod')
api_key = resp['SecretString']

Vault read via CLI:

vault kv get -field=api_key secret/llm/openai

If you operate a multi-model gateway (e.g., n4n.ai, which exposes one OpenAI-compatible endpoint for 240+ models), you’ll cache these reads in-memory; neither service is designed for per-inference latency budgets.

Rotation

AWS rotation requires writing a Lambda and configuring a schedule through the console or Terraform. The Lambda must implement create_secret, set_secret, test_secret, and finish_secret steps. Vault supports native secret rotation with custom plugins or the UI, but you still script the provider call and handle rollback. Both approaches are scriptable; AWS bundles the orchestration, Vault gives you the primitives.

Price/Cost Model

AWS Secrets Manager charges $0.40 per secret per month and $0.05 per 10,000 API calls (public pricing). A fleet of 50 provider keys costs $20/mo plus call volume. If you mistakenly fetch every secret on every LLM request without caching, API call costs dwarf the storage fee. Vault open source is free to run, but you pay for the EC2/EKS nodes, backups, and engineering time to operate it. Vault Enterprise adds per-node licensing with SLA and advanced replication.

The aws secrets manager vs vault api keys cost debate flips at scale: managed fees grow linearly with secret count, while self-hosted cost is dominated by infrastructure and headcount. At 1,000 secrets, AWS is ~$400/mo before calls; a small Vault cluster on EKS might be $200/mo in nodes plus ops.

Latency/Throughput

A direct AWS SM GetSecretValue call typically lands in 10–30 ms within the same region. Vault over local network is similar when backed by Raft or Consul on healthy nodes. Both throttle: AWS has account-level TPS limits (default burst 2,000 TPS); Vault depends on storage backend and leader election. For LLM apps, fetch keys at startup or on a TTL cache (e.g., 5 minutes). Never block a chat completion on a secrets API call.

import time, boto3
_cache = {}
def get_key(secret_id, ttl=300):
    now = time.time()
    if secret_id in _cache and now - _cache[secret_id][1] < ttl:
        return _cache[secret_id][0]
    val = boto3.client('secretsmanager').get_secret_value(SecretId=secret_id)['SecretString']
    _cache[secret_id] = (val, now)
    return val

Cross-region replication adds latency if you read from a secondary, but both systems support regional endpoints.

Ergonomics

AWS SM ships first-class SDKs in boto3, AWS SDK for Go, and others, plus a native Terraform resource aws_secretsmanager_secret. IAM policies grant least privilege per secret ARN. Vault uses the vault CLI, official client libraries, and a Kubernetes Auth method with a sidecar injector that writes secrets to a tmpfs volume. The injector pattern removes the need to call the API from app code.

Terraform for Vault KV:

resource "vault_mount" "llm" {
  path = "llm"
  type = "kv-v2"
}

AWS IAM policy snippet limiting read to LLM secrets:

{
  "Effect": "Allow",
  "Action": "secretsmanager:GetSecretValue",
  "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:openai-*"
}

Vault’s UI is serviceable; AWS Console is fine for occasional manual rotation. Debugging access denials in Vault requires reading audit logs; in AWS, CloudTrail shows the exact principal and policy.

Ecosystem

AWS SM lives inside the AWS boundary: CloudTrail logs every access, cross-account replication via replica regions, and VPC endpoints keep traffic private. Vault plugs into Okta, GitHub, Kubernetes, and any cloud; it speaks AppRole, OIDC, and LDAP, and can act as a broker for cloud credentials. If your stack is all-in on AWS, the aws secrets manager vs vault api keys choice leans toward SM. Multi-cloud or bare-metal favors Vault.

Vault’s secret engines extend to databases, SSH, and PKI—useful if your LLM platform also manages Postgres credentials. AWS SM stays focused on secrets and parameter store overlap.

Limits

AWS Secrets Manager caps secret size at 64 KB and max 100,000 secrets per account (soft limit, raiseable). Request throughput is bounded by the service; sustained above 2,000 TPS requires a support ticket. Vault limits are backend-dependent; Raft storage handles thousands of reads/sec on modest nodes, but a misconfigured Consul cluster will bottleneck. Vault open source has no hard secret count cap, but memory grows with audit logging.

Comparison Table

Which to Choose

All-in AWS, small team: Use AWS Secrets Manager. IAM and CloudTrail remove boilerplate. The $0.40/secret fee is negligible at <100 keys. You avoid running a stateful system.

Multi-cloud or compliance-heavy: Vault open source on EKS or bare metal. You control encryption keys, replication, and can enforce complex auth. Expect to own backups and upgrades.

High-frequency LLM gateway: Either works if you cache. If you front 240+ models like n4n.ai, store provider keys in Vault or SM but read them once per process and rotate via signal or short TTL. Don’t pay per-call taxes on every token.

Strict cost control at scale: Self-hosted Vault wins past ~500 secrets when the AWS line item grows linearly. Factor engineering time—Vault isn’t free if a human must babysit it.

Need dynamic cloud creds too: Vault’s AWS/GCP dynamic roles outweigh SM’s static store. If you already run Vault for DB credentials, put LLM keys there to consolidate.

Pick based on where your infrastructure already lives and how much operational burden you’ll accept for control.