Setting per-key rate limits and spend caps

When you expose an LLM endpoint to multiple clients, you need per-key rate limits spend caps api controls to stop a single bad actor or buggy loop from exhausting your quota or blowing the bill. This guide walks through building those controls at the gateway layer with Redis and a thin proxy, using OpenAI-compatible responses so the logic ports to any provider. You will issue keys, throttle requests per minute and per token, and hard-cut at a spend ceiling.

Step 1: Define the key policy schema

Start by deciding what each API key can do. At minimum you need request rate, token rate, and a spend ceiling. Store these as signed claims or in a fast lookup store.

{
  "key_id": "k_3f9a2b",
  "rpm_limit": 60,
  "tpm_limit": 50000,
  "spend_cap_usd": 20.00,
  "model_allowlist": ["gpt-4o-mini", "mistral-7b"],
  "created_at": "2025-01-15T10:00:00Z"
}

Keep the mutable counters (used_rpm, used_tpm, spent_usd) in Redis with TTLs matching your window. The static policy lives in your database; the hot counters live in memory.

Step 2: Issue keys and persist the policy

Generate a high-entropy secret, store a hash, and write the policy. Never log the raw key after creation.

import secrets, hashlib, redis, json

r = redis.Redis()

def create_key(rpm=60, tpm=50000, cap=20.0):
    raw = "sk-" + secrets.token_urlsafe(24)
    key_id = "k_" + hashlib.sha256(raw.encode()).hexdigest()[:8]
    policy = {
        "key_id": key_id,
        "rpm_limit": rpm,
        "tpm_limit": tpm,
        "spend_cap_usd": cap,
        "model_allowlist": ["gpt-4o-mini"]
    }
    r.hset(f"policy:{key_id}", mapping=policy)
    r.set(f"counter:{key_id}:rpm", 0, ex=60)
    r.set(f"counter:{key_id}:tpm", 0, ex=60)
    r.set(f"spent:{key_id}", "0.0")
    return raw, key_id

Hand the raw string to the client once. Your gateway resolves it to key_id on every request via an auth header.

Step 3: Enforce rate limits with an atomic token bucket

Use a Lua script so the increment and check happen atomically. This avoids race conditions when many requests arrive at once.

-- rate_limit.lua
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local cost = tonumber(ARGV[2])
local ttl = tonumber(ARGV[3])

local used = redis.call("INCRBY", key, cost)
if used == cost then
  redis.call("EXPIRE", key, ttl)
end
if used > limit then
  return 0
end
return 1

Call it from Python before forwarding:

def allow(key_id, cost, limit, ttl, counter_prefix):
    script = r.register_script(RATE_LIMIT_LUA)
    return script(keys=[f"{counter_prefix}:{key_id}"],
                  args=[limit, cost, ttl]) == 1

if not allow(key_id, 1, policy["rpm_limit"], 60, "counter:rpm"):
    return {"error": "rate limited"}, 429

Do the same for tpm using the prompt+completion token estimate from the request body, or after the fact using the response.

Step 4: Meter spend from provider usage

After a successful completion, the OpenAI-compatible response includes a usage object. Multiply token counts by your known per-model prices. If you route through n4n.ai, its per-token usage metering returns exact token counts in that object, so your cost math stays precise without re-tokenizing.

PRICES = {  # example only, replace with your contracted rates
    "gpt-4o-mini": {"prompt": 0.00015, "completion": 0.0006},
}

def record_spend(key_id, model, usage):
    price = PRICES.get(model, {"prompt": 0.001, "completion": 0.001})
    cost = usage["prompt_tokens"] * price["prompt"] \
         + usage["completion_tokens"] * price["completion"]
    r.incrbyfloat(f"spent:{key_id}", cost)

Run this in the response path of your proxy. If the upstream supports streaming, sum deltas at the end.

Step 5: Cut off over-budget keys

Before proxying, check spent against cap. Do it atomically to avoid a burst slipping through.

def within_budget(key_id, cap):
    spent = float(r.get(f"spent:{key_id}") or 0.0)
    return spent < cap

if not within_budget(key_id, policy["spend_cap_usd"]):
    return {"error": "spend cap reached"}, 402

Pair this with a background job that zeros counters on a billing cycle. For hard caps, also revoke the key in the policy store so subsequent auth fails fast.

Step 6: Verify the controls end to end

Issue a test key with low limits, then hammer it.

export TEST_KEY=$(python -c "from keys import create_key; print(create_key(5, 100, 0.01)[0])")
for i in $(seq 1 10); do
  curl -s -o /dev/null -w "%{http_code}\n" \
    -H "Authorization: Bearer $TEST_KEY" \
    -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}' \
    https://api.your-gateway.com/v1/chat/completions
done

You should see 200 for the first five, then 429. To test spend, set spend_cap_usd to 0.001 and send a few larger prompts; you will get 402 after the meter crosses the line.

Check Redis directly:

redis-cli get spent:k_3f9a2b
redis-cli get counter:k_3f9a2b:rpm

If the numbers move and the status codes match, the per-key rate limits spend caps api logic is working.

Production caveats

Rate limit windows should be sliding, not fixed, to prevent the stampede at the top of each minute. Use a sorted set if you need accuracy. Spend caps must account for token price variance across models; never assume a single rate. And always return Retry-After headers so clients back off correctly.

The pattern above is provider-agnostic. Whether you front a single vendor or an OpenAI-compatible gateway with fallback, the auth and metering layer stays the same. Build it once, then tune limits per tenant from real traffic.