Setting per-key rate limits for multi-tenant LLM apps

Multi-tenant LLM applications break down when one customer’s traffic floods the shared inference pool. Implementing per-key rate limits multi-tenant llm deployments is the only scalable way to keep noisy neighbors from blowing your provider quota or bankrupting your token budget. This guide walks through a concrete architecture you can ship this week, from key issuance to verified enforcement.

Step 1: Model your tenants and API keys

You need a stable mapping from issued key to tenant ID and plan tier. Do not embed tenant info in the key itself unless you sign it with HMAC. A simple approach stores a key prefix in Redis or Postgres:

# issuance.py
import secrets, redis

r = redis.Redis()

def create_key(tenant_id: str, tier: str) -> str:
    key = "sk-" + secrets.token_urlsafe(24)
    # store only prefix to avoid plaintext key at rest
    r.hset(f"tenant:key:{key[:10]}", mapping={"tenant": tenant_id, "tier": tier})
    return key

The first 10 characters are enough to route lookups without exposing the full secret. Define tier limits explicitly:

TIERS = {
    "free": {"req_per_sec": 0.1, "req_cap": 5, "tok_per_sec": 100, "tok_cap": 5000},
    "pro":  {"req_per_sec": 1.0, "req_cap": 60, "tok_per_sec": 2000, "tok_cap": 100000},
}

These numbers are examples; set them from your provider’s actual quotas and your margin.

Step 2: Choose a rate limit algorithm

For LLM gateways, a sliding window log is precise but memory-heavy. A token bucket is simpler, tolerates bursts, and maps cleanly to “tokens per minute” semantics. Implement it in Redis with a Lua script so the check-and-decrement is atomic across workers.

-- token_bucket.lua
local key = KEYS[1]
local now = tonumber(ARGV[1])
local rate = tonumber(ARGV[2])      -- tokens per second
local capacity = tonumber(ARGV[3])
local requested = tonumber(ARGV[4])

local data = redis.call("HMGET", key, "tokens", "ts")
local tokens = tonumber(data[1]) or capacity
local ts = tonumber(data[2]) or now

local delta = (now - ts) * rate
if delta < 0 then delta = 0 end
tokens = math.min(capacity, tokens + delta)

if tokens >= requested then
  tokens = tokens - requested
  redis.call("HMSET", key, "tokens", tokens, "ts", now)
  return 1
else
  redis.call("HMSET", key, "tokens", tokens, "ts", now)
  return 0
end

Load the script once at startup and call it via EVALSHA. This gives you per-key rate limits multi-tenant llm requests can’t bypass by hitting multiple app instances.

Step 3: Build enforcement middleware

Wrap your OpenAI-compatible endpoint. Extract the key, map to tenant, run the bucket for request count and estimated token cost (use max_tokens as a ceiling).

# middleware.py
from fastapi import Request, Response
import redis, time, hashlib

r = redis.Redis()
BUCKET_SHA = r.script_load(open("token_bucket.lua").read())

async def rate_limit(request: Request, call_next):
    auth = request.headers.get("Authorization", "")
    key = auth.replace("Bearer ", "").strip()
    if not key.startswith("sk-"):
        return Response("invalid key", status_code=401)
    
    meta = r.hgetall(f"tenant:key:{key[:10]}")
    if not meta:
        return Response("unknown key", status_code=401)
    tenant = meta[b"tenant"].decode()
    tier = meta[b"tier"].decode()
    cfg = TIERS[tier]
    
    body = await request.json()
    est_tokens = body.get("max_tokens", 1000)
    
    now = int(time.time())
    ok_req = r.evalsha(BUCKET_SHA, 1, f"rl:{tenant}:req", now,
                       cfg["req_per_sec"], cfg["req_cap"], 1)
    ok_tok = r.evalsha(BUCKET_SHA, 1, f"rl:{tenant}:tok", now,
                       cfg["tok_per_sec"], cfg["tok_cap"], est_tokens)
    
    if not ok_req or not ok_tok:
        return Response(
            '{"error":"rate limited"}',
            status_code=429,
            media_type="application/json",
            headers={"X-RateLimit-Remaining": "0"}
        )
    
    response = await call_next(request)
    return response

This pattern enforces per-key rate limits multi-tenant llm traffic at the edge of your service before you spend a single upstream token.

Step 4: Account for actual token usage post-response

Estimates are not enough—a request can return far fewer tokens than max_tokens. After the upstream responds, deduct the real prompt+completion tokens from the tenant’s bucket.

# inside call_next wrapper
resp_body = await response.body()
try:
    usage = json.loads(resp_body).get("usage", {})
    used = usage.get("total_tokens", 0)
except:
    used = est_tokens

# refund the overestimate, then charge actual
r.evalsha(BUCKET_SHA, 1, f"rl:{tenant}:tok", int(time.time()),
          cfg["tok_per_sec"], cfg["tok_cap"], est_tokens)  # add back
r.evalsha(BUCKET_SHA, 1, f"rl:{tenant}:tok", int(time.time()),
          cfg["tok_per_sec"], cfg["tok_cap"], -used)       # subtract real

If you route through n4n.ai, its per-token usage metering already returns accurate totals on every response, so you can skip the estimation step and reconcile directly from the metered value.

Step 5: Handle provider 429s and fallback

Local limits do not guarantee the upstream provider won’t rate-limit you. Implement exponential backoff and respect Retry-After. If your gateway supports automatic fallback when a provider is degraded, your per-key logic still runs before the request leaves your network.

import asyncio, httpx

async def forward_with_retry(payload, headers, url):
    backoff = 1
    for _ in range(4):
        resp = await httpx.AsyncClient().post(url, json=payload, headers=headers)
        if resp.status_code == 429:
            retry_after = resp.headers.get("Retry-After")
            await asyncio.sleep(int(retry_after) if retry_after else backoff)
            backoff *= 2
            continue
        return resp
    return Response("upstream unavailable", status_code=503)

A circuit breaker per tenant prevents a stuck client from hammering retry loops.

Step 6: Surface limit headers to clients

Clients need to see their remaining quota. Compute remaining from the bucket state and return standard headers.

remaining = int(r.hget(f"rl:{tenant}:req", "tokens") or 0)
response.headers["X-RateLimit-Remaining"] = str(remaining)
response.headers["X-RateLimit-Limit"] = str(cfg["req_cap"])
response.headers["X-RateLimit-Reset"] = str(int(time.time()) + 60)

Return a JSON error body with type: rate_limit_exceeded so SDKs parse it correctly. This makes per-key rate limits multi-tenant llm consumption debuggable for your users.

Step 7: Verify the setup

Spin up Redis and the FastAPI app. Issue two keys for different tenants. Use a loop to exceed the free tier.

# create keys (pseudo)
python issuance.py > keys.txt

# load test with curl
for i in {1..10}; do
  curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:8000/v1/chat/completions \
    -H "Authorization: Bearer $(head -1 keys.txt)" \
    -d '{"model":"gpt-3.5-turbo","max_tokens":50,"messages":[{"role":"user","content":"hi"}]}'
done

Expect the first 5 requests to return 200 and the rest 429 with X-RateLimit-Remaining: 0. Check Redis:

redis-cli HGETALL rl:tenant1:req

You should see tokens near zero and ts within the last second.

For automated verification, add a pytest fixture:

def test_rate_limit(client, free_key):
    codes = [client.post("/v1/chat/completions",
             headers={"Authorization": f"Bearer {free_key}"},
             json={"max_tokens":10,"messages":[]}).status_code
             for _ in range(7)]
    assert codes[:5] == [200]*5
    assert set(codes[5:]) == {429}

Operational notes

Persist tenant tiers in a config service so you can change limits without redeploy. Use a separate bucket for cost (dollars) if you resell access. Monitor 429 rates per tenant; a spike means a misconfigured client or a brute-force attempt.

Per-key rate limits multi-tenant llm systems are not a nice-to-have. They are the boundary that keeps your inference bill predictable and your latency p99 flat when one tenant decides to batch 10k summarizations at midnight. Build the bucket once, wire it into middleware, and let Redis do the counting.