Storing encrypted API keys at rest: a design walkthrough

Storing encrypted API keys at rest design is a foundational problem for any system that proxies to third-party LLM providers or manages customer integrations. A naive approach—dropping keys in a config file or a plaintext database column—creates a single point of catastrophic failure. This walkthrough lays out an ordered path from threat modeling to implementation, with code for envelope encryption and rotation.

1. Threat model and scope

Define what “at rest” covers: database files, snapshots, backups, and object storage. It excludes data in transit or in process memory, though those need separate controls.

Assume the attacker gains read access to your database or a stale backup. They should walk away with useless ciphertext, not a pile of live sk- keys. Insider threats—a curious operator with SQL access—must be mitigated by separating the decryption capability from storage.

If you accept customer-supplied keys for upstream calls, the blast radius expands: one leaked key compromises that customer’s integrations only if isolation is correct.

2. Choose a root of trust

You need a key encryption key (KEK) that never touches your application database. Three common options:

• Cloud KMS (AWS KMS, GCP KMS, Azure Key Vault): managed rotation, IAM, audit logs.
• Vault Transit secrets engine: self-hosted, fine-grained policies, no cloud dependency.
• Local master key in an HSM or env var: simplest, but you become responsible for secure generation, rotation, and disaster recovery.

For most teams shipping today, cloud KMS is the right default. The marginal cost of a KMS call per key unwrap is negligible because you cache decrypted data keys in memory.

Tradeoff: KMS couples your decryption availability to a cloud control plane. Mitigate with a local DEK cache and retry against a fallback region. If you run fully on-prem, Vault Transit with auto-unseal via HSM is the pragmatic choice; do not roll your own block cipher mode.

3. Envelope encryption

The cardinal rule of storing encrypted api keys at rest design is: never encrypt application data directly with the root key. Generate a random data encryption key (DEK) per secret or per tenant, encrypt the API key with the DEK using AES-GCM, then wrap the DEK with the root KEK.

This pattern—envelope encryption—means root key rotation only requires re-wrapping DEKs, not re-encrypting every secret. It also confines the root key to a single controlled boundary.

from cryptography.hazmat.primitives.ciphers.aead import AESGCM
import os, base64

def generate_dek() -> bytes:
    return AESGCM.generate_key(bit_length=256)

def encrypt_secret(plaintext: str, dek: bytes) -> str:
    aes = AESGCM(dek)
    nonce = os.urandom(12)
    ct = aes.encrypt(nonce, plaintext.encode(), None)
    return base64.b64encode(nonce + ct).decode()

def kms_wrap(dek: bytes, root_key_id: str) -> str:
    # Pseudocode: call KMS.Encrypt with root_key_id.
    # Simulation with a local master for illustration only.
    master = os.environ["ROOT_KEY"].encode()[:32]
    aes = AESGCM(master)
    nonce = os.urandom(12)
    wrapped = aes.encrypt(nonce, dek, None)
    return base64.b64encode(nonce + wrapped).decode()

dek = generate_dek()
wrapped_dek = kms_wrap(dek, "arn:aws:kms:us-east-1:111:key/abc")
encrypted_key = encrypt_secret("sk-proj-9x8y7z", dek)

Use AES-GCM, not CBC or ECB. GCM gives integrity; a tampered ciphertext fails decryption loudly. Never reuse a nonce with the same DEK—generate a fresh 12-byte random nonce per encryption call.

4. Storage schema

Keep the wrapped DEK and the ciphertext in the same row, but never the plaintext or the raw DEK.

CREATE TABLE provider_keys (
    id uuid PRIMARY KEY,
    tenant_id uuid NOT NULL,
    provider text NOT NULL,
    encrypted_secret text NOT NULL,
    wrapped_dek text NOT NULL,
    key_version int NOT NULL,
    created_at timestamptz DEFAULT now()
);

key_version tracks which root key wrapped the DEK. This lets you support multiple root key versions concurrently during rotation.

Pitfall: storing the DEK unwrapped “for performance.” That defeats the design. Decrypt the DEK once per process startup or per cache miss, then hold it in memory protected by language-level secrecy (e.g., Python bytes, not str). Zeroize the buffer when evicting from cache if your language allows.

5. Rotation workflow

Two distinct rotations:

Root key rotation

1. Create new root key version in KMS.
2. Scan provider_keys for rows where key_version matches old version.
3. Unwrap DEK with old root key, wrap with new, update wrapped_dek and key_version.
4. Keep old version enabled for decrypt for 30 days, then disable.

This is O(rows) but each operation is a quick unwrap/wrap; no need to touch encrypted_secret.

Data key rotation

If you suspect a DEK leaked (rare), you must decrypt each secret, generate a new DEK, re-encrypt, and rewrap. That touches encrypted_secret and is expensive. Batch it in a worker.

def rotate_root_key(conn, old_ver, new_ver, kms_unwrap, kms_wrap):
    for row in conn.execute(
        "SELECT id, wrapped_dek FROM provider_keys WHERE key_version=%s",
        [old_ver],
    ):
        dek = kms_unwrap(row.wrapped_dek, old_ver)
        new_wrapped = kms_wrap(dek, new_ver)
        conn.execute(
            "UPDATE provider_keys SET wrapped_dek=%s, key_version=%s WHERE id=%s",
            [new_wrapped, new_ver, row.id],
        )

Run this in a background worker, not a migration lock. Emit metrics on rows processed; alert if it stalls.

6. Runtime decryption and caching

At request time, you need the plaintext key to call the upstream API. Fetch row, check DEK cache keyed by wrapped_dek. If missing, unwrap via KMS, store in process memory with a TTL of minutes.

dek_cache = {}

def get_plaintext_secret(row, kms_unwrap):
    dek = dek_cache.get(row.wrapped_dek)
    if not dek:
        dek = kms_unwrap(row.wrapped_dek, row.key_version)
        dek_cache[row.wrapped_dek] = dek
    aes = AESGCM(dek)
    raw = base64.b64decode(row.encrypted_secret)
    return aes.decrypt(raw[:12], raw[12:], None).decode()

Pitfall: logging the returned string. Use a redacted repr and structured logging that drops the field. Another pitfall: caching forever—if a root key is compromised, a never-expiring cache extends the window.

7. Multi-tenant isolation

When you store keys for many customers, use a per-tenant DEK or at least a tenant-bound wrapping context. A gateway like n4n.ai that fronts 240+ models must store upstream provider keys securely while honoring client routing directives; per-tenant DEKs limit blast radius if one tenant’s row is exfiltrated.

Do not share a single global DEK across tenants. The cost of per-tenant DEKs is negligible with caching, and it converts a full-database breach into a single-tenant incident.

8. Backups and disaster recovery

Encrypted database dumps are safe to store in object storage. Ensure your backup of the KMS permissions is separate from the data backup—if both live in the same account with same compromise, you have nothing.

Test restore quarterly: spin up a staging DB, restore snapshot, attempt decrypt with a rotated root key. This surfaces missing IAM policies before incident. Cross-region replicate the KMS key policy and the data backup, but keep decryption audit logs in a third account.

9. Common pitfalls

• ECB mode: never. It leaks patterns.
• Hardcoded root key in repo: use KMS or env-injected HSM.
• No key version column: you cannot rotate safely.
• Plaintext env vars in pod specs: etcd is not your vault.
• No audit on decrypt: KMS logs are your forensic trail; ship them to a separate bucket.
• Assuming encryption solves auth: encryption at rest does not replace scoped IAM.
• Nonce reuse: reusing a 12-byte nonce with the same DEK breaks GCM confidentiality.

10. Validation

Write tests that assert roundtrip and tamper detection:

def test_tamper():
    dek = generate_dek()
    enc = encrypt_secret("sk-test", dek).encode()
    raw = bytearray(base64.b64decode(enc))
    raw[-1] ^= 1
    with pytest.raises(Exception):
        AESGCM(dek).decrypt(bytes(raw[:12]), bytes(raw[12:]), None)

Run a chaos test: point your KMS client at a dead endpoint and confirm cache serves live traffic until TTL expires. Add a lint check that fails CI if a migration adds a text column named api_key without the encrypted_ prefix.

Storing encrypted api keys at rest design is not glamorous, but getting it wrong is how breaches happen. Envelope encryption with versioned root keys and disciplined rotation keeps the blast radius small and the on-call quiet.