n4nAI
Open console →
API integrationHow-toAWS Lambda Serverless LLM Integration

Storing provider API keys securely in AWS Secrets Manager

Learn how to store and retrieve AWS Secrets Manager LLM API keys for Lambda-based inference, with IAM policies, caching, and rotation steps.

n4n Team5 min read1,087 words

Audio narration

Coming soon — every post will get a voice note here.

Hardcoding LLM provider credentials in a Lambda deployment package or environment configuration exposes them to anyone with read access to the function or its CloudFormation template. Using aws secrets manager llm api keys moves those credentials into a managed store with per-secret IAM policies, automatic encryption, and rotation hooks. This guide walks through a concrete setup that survives production traffic without leaking keys into logs or images.

Step 1: Decide on a secret layout

You rarely have just one key. OpenAI, Anthropic, Mistral, and a gateway such as n4n.ai each issue separate credentials. Two patterns work in AWS Secrets Manager: one secret per provider, or a single JSON secret containing a map. For Lambda workloads, a single secret reduces API calls and simplifies caching, at the cost of broader blast radius if the blob is misread.

Create a local file:

{
  "openai": "sk-...",
  "anthropic": "sk-ant-...",
  "n4n": "nk-...",
  "cohere": "sk-..."
}

Then push it:

aws secretsmanager create-secret \
  --name prod/llm/provider-keys \
  --secret-string file://llm-keys.json \
  --region us-east-1 \
  --tags Key=service,Value=llm-inference

If you prefer isolation, replace with --name prod/llm/openai and store the raw string. Either way, aws secrets manager llm api keys should never be committed to Git or baked into a layer. Tag the secret for cost allocation and access reviews; untagged secrets accumulate and become invisible in audits.

For multi-region resilience, replicate the secret to a secondary region with replicate-secret-to-regions. Your Lambda in us-west-2 can then read the local copy instead of crossing the continent on every cold start.

Step 2: Scope IAM permissions to the Lambda role

The function’s execution role needs secretsmanager:GetSecretValue on that exact ARN. Avoid Resource: "*" even in dev—over-broad policies are how a compromised function becomes a credential dump.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/llm/provider-keys-*"
    }
  ]
}

If you use a customer-managed KMS key instead of the default, the role also needs kms:Decrypt on that key ARN. The KMS key policy must permit the Lambda role; IAM alone is insufficient.

Attach this to the role already assumed by your Lambda. If the function runs in a VPC, confirm the Secrets Manager endpoint is reachable—either via NAT or a VPC interface endpoint. Missing networking is the most common cause of timeout on the first call, and it surfaces as a generic EndpointConnectionError, not an auth failure.

Step 3: Retrieve and cache the secret in Lambda

Cold starts should fetch once and reuse across invocations in the same execution context. Store the parsed dict in a module-level variable. The execution context can stay warm for hours; repeated GetSecretValue calls waste quota and add latency.

import boto3
import json
import os

SECRET_NAME = os.environ.get("LLM_SECRET_NAME", "prod/llm/provider-keys")
REGION = os.environ.get("AWS_REGION", "us-east-1")

_client = boto3.client("secretsmanager", region_name=REGION)
_cache = None

def get_llm_keys():
    global _cache
    if _cache is not None:
        return _cache
    try:
        resp = _client.get_secret_value(SecretId=SECRET_NAME)
    except _client.exceptions.ResourceNotFoundException:
        raise RuntimeError(f"Secret {SECRET_NAME} missing in {REGION}")
    _cache = json.loads(resp["SecretString"])
    return _cache

This code decrypts via the default KMS key or your customer-managed key automatically. Do not log the returned dict. If you need to debug, log len(v) per key. Wrap the call with a short retry using tenacity or botocore.retries if you see transient throttling at scale.

Step 4: Inject keys into the LLM client

Most Python SDKs accept the key at construction. For direct provider calls:

from openai import OpenAI

def handler(event, context):
    keys = get_llm_keys()
    client = OpenAI(api_key=keys["openai"])
    # use client.chat.completions.create(...)

If you consolidate routing behind a single OpenAI-compatible gateway, you only need one entry from aws secrets manager llm api keys. For example, pointing at n4n.ai means storing nk-... and setting base_url="https://api.n4n.ai/v1"; the gateway forwards to 240+ models and handles provider fallback when a backend is degraded. The retrieval code above does not change.

client = OpenAI(
    api_key=keys["n4n"],
    base_url="https://api.n4n.ai/v1"
)

That pattern cuts the number of secrets you rotate and the surface area for leakage. If you use an async handler, construct AsyncOpenAI once per container instead of per event to reuse the connection pool.

Step 5: Rotate without downtime

Secrets Manager supports rotation through a separate Lambda. For LLM keys, providers rarely support a two-key overlap, so a custom rotation function that calls the provider’s key-roll API and updates the secret is preferable. At minimum, schedule a manual rotation every 90 days:

aws secretsmanager update-secret \
  --secret-id prod/llm/provider-keys \
  --secret-string file://new-llm-keys.json

Because your function caches the secret for the life of the execution context, a slow rollout is safe: old containers keep old keys until they’re recycled. Force a fleet refresh by deploying a no-op code change or calling UpdateFunctionCode if you need immediate consistency. For automated rotation, use the secretsmanager-rotation blueprint and guard against partial provider failures by writing the new secret only after all provider key-roll endpoints return 200.

Step 6: Verify the setup end to end

Two checks matter: the function can read the secret, and the key actually authenticates to the provider.

First, invoke the function with a test event that returns masked metadata:

def test_handler(event, context):
    keys = get_llm_keys()
    return {k: len(v) for k, v in keys.items()}

Expected response: {"openai": 51, "anthropic": 108, ...}. If you get AccessDenied, the IAM resource ARN is wrong or the KMS key policy blocks the role. If you get a timeout, check VPC routing or the region mismatch.

Second, make a minimal provider call:

def verify_handler(event, context):
    keys = get_llm_keys()
    client = OpenAI(api_key=keys["openai"])
    models = client.models.list()
    return {"count": len(models.data)}

A 200 with a count confirms the aws secrets manager llm api keys are live and correctly wired. In CloudWatch, confirm no plaintext key appears in logs—CloudTrail will show GetSecretValue from your Lambda role, not from your own CLI user. If you see the key in a log line, add a log filter or redaction policy before shipping to a centralized sink.

Operational caveats

  • Region mismatch: The secret and the Lambda must share a region unless you explicitly call cross-region. Boto3 does not infer across regions, and the error message will not say “wrong region” clearly.
  • Cache invalidation: If you rotate and immediately need the new key, null out _cache via an environment flag or request a specific VersionId. The default AWSCURRENT stage updates only after the new version is labeled.
  • Concurrent cold starts: Multiple fresh containers may hit Secrets Manager simultaneously. The per-account quota is generous but not infinite; if you scale from zero to thousands of concurrent invocations, pre-warm or use the Parameters and Secrets Lambda Extension.
  • Extension alternative: The Parameters and Secrets Lambda Extension runs inside the execution environment and caches with a TTL, removing the need for manual _cache. Configure it via layer ARN and environment variables.
{
  "AWS_LAMBDA_EXEC_WRAPPER": "/opt/extensions/aws-parameters-and-secrets-lambda-extension",
  "SECRETS_MANAGER_TIMEOUT_MILLIS": "5000"
}

Then read from http://localhost:2773/secretsmanager/get?secretId=prod/llm/provider-keys inside the function. This offloads caching and respects the extension’s built-in KMS auth, and it survives container reuse without custom globals.

Step 7: Clean up local files

After creating or updating the secret, shred the local JSON. It is now in AWS; a stray copy in your home directory is the exact risk you tried to eliminate.

shred -u llm-keys.json new-llm-keys.json

If you must keep them for disaster recovery, use git-crypt or a password manager, never the repo. Treat the local file as a temporary transport, not a source of truth.

Wrapping up

Treating aws secrets manager llm api keys as code artifacts leads to leaked credentials and failed audits. Push them to Secrets Manager, scope IAM tightly, cache in the Lambda global scope or the official extension, and verify with a masked-length test before trusting production traffic. The pattern takes an afternoon to implement and removes a whole class of incident from your on-call rotation.

Tagsaws-lambdasecrets-managersecurityapi-keys

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
API integrationAWS Lambda vs EC2 for hosting LLM API integrationsA pragmatic engineer's comparison of AWS Lambda vs EC2 for LLM API hosting across cost, latency, limits, and ergonomics, with a use-case verdict.API integrationBuilding an AWS Lambda webhook handler for LLM function callingStep-by-step tutorial for building an AWS Lambda webhook handler that uses LLM function calling with Python, API Gateway, and deployable code.API integrationDeploying an AWS Lambda LLM proxy with the Serverless FrameworkHands-on guide to building and deploying an aws lambda serverless framework llm proxy that forwards LLM requests to OpenAI-compatible endpoints via API Gateway.API integrationRate limiting LLM calls from AWS Lambda with SQS queuesPractical guide to aws lambda sqs rate limiting llm calls: use SQS visibility timeouts, reserved concurrency, and batching to stay under provider limits.API integrationAWS Lambda cold starts and LLM API latency: what to expectAnalyze how AWS Lambda cold starts affect LLM API latency, where the time actually goes, and practical patterns to keep serverless inference responsive.API integrationAWS Lambda function URLs for low-latency LLM proxyingLearn how to build an aws lambda function urls llm proxy for low-latency LLM inference, with step-by-step setup, code, and verification.

All aws lambda serverless llm integration posts →