Moving from per-provider rate limits to unified quota is the first real step toward operating LLMs like a utility instead of a fleet of temperamental APIs. Today you likely call OpenAI, Anthropic, and maybe Gemini or Mistral directly, each enforcing its own requests-per-minute and tokens-per-minute caps, returning 429s on different axes, and degrading independently. This guide lays out an ordered path to collapse those per-provider rate limits to unified quota behind a single OpenAI-compatible gateway, so your application code sees one budget and one error surface.
1. Inventory existing provider constraints
Before you change anything, write down exactly what each provider enforces. Most vendors publish tier-based request-per-minute (RPM) and token-per-minute (TPM) limits; some separate write and read tokens. Capture your peak observed traffic per model and per endpoint from the last 30 days.
# limits.py - snapshot of current state
PROVIDERS = {
"openai": {"rpm": "tier-dependent", "tpm": "tier-dependent", "models": ["gpt-4o", "gpt-4o-mini"]},
"anthropic": {"rpm": "account-based", "tpm": "account-based", "models": ["claude-3-5-sonnet", "claude-3-haiku"]},
"gemini": {"rpm": "project-based", "tpm": "project-based", "models": ["gemini-1.5-pro"]},
}Pull the actual usage field from past responses to compute true token throughput. A single large document summarization can blow your TPM budget while RPM stays near zero. Convert historical traffic to token counts per minute before proceeding—this is the only number that survives cross-provider translation.
Pitfall: mixing axes
Teams often set alerts on RPM because it is easy to count. If your traffic is token-heavy (long contexts, agentic loops), RPM is a vanity metric. Standardize on TPM from day one.
2. Route all traffic through one endpoint
Replace direct SDK initialization with a single base URL. The OpenAI Python client works against any OpenAI-compatible server, so you avoid rewriting call sites. Do the same for TypeScript services.
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.example.com/v1", # your unified gateway
api_key="your-gateway-key",
)
resp = client.chat.completions.create(
model="anthropic/claude-3-5-sonnet", # provider prefixed or mapped
messages=[{"role": "user", "content": "Summarize this RFC."}],
)// ts equivalent
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://gateway.example.com/v1",
apiKey: process.env.GW_KEY,
});Tradeoff: you add one network hop. In practice the gateway is a thin proxy; p99 latency increase is single-digit milliseconds if colocated. Do not skip this step by building your own retry layer—you will reimplement quota logic badly and ship bugs under pressure.
3. Express quota in token terms, not requests
Unified quota should be a token bucket, because tokens are the common denominator across providers and model families. Define a global tokens-per-minute (TPM) cap for your org, then subdivide by team or route.
{
"quota": {
"global_tpm": 200000,
"per_api_key": {
"backend-svc": 50000,
"eval-jobs": 20000
}
}
}If you previously reasoned in RPM, approximate: tpm ≈ rpm * avg_tokens_per_request. Measure average completion + prompt tokens from logs to calibrate. A gateway with per-token usage metering lets you enforce this bucket without trusting client-reported counts. Implement the bucket as a leaky meter: tokens leave the bucket at the provider’s real rate, but your gateway rejects when the local bucket is empty, not when the provider returns 429.
Subdividing safely
Never give a single API key the full global quota. Leave 20% headroom for fallback bursts (see next section). Otherwise a fallback storm consumes the entire org budget.
4. Implement fallback and priority routing
When one provider is degraded or rate-limited, requests should shift to an equivalent model. This is where the move from per-provider rate limits to unified quota pays off: a 429 from Anthropic no longer means a failed user request, just a routing decision.
# routing directive sent by client
headers = {
"x-n4n-routing": "fallback:openai/gpt-4o,gemini/gemini-1.5-pro",
}A gateway that honors client routing directives will try the primary, then fall back automatically. n4n.ai, for instance, provides one OpenAI-compatible endpoint covering 240+ models and performs automatic fallback when a provider is rate-limited or degraded, collapsing those per-provider rate limits to unified quota without custom code. You still need to set timeouts—fallback adds tail latency.
Pitfall: silent quality drift
Fallback to a different model changes output format. Constrain fallbacks to capability-equivalent models and test parsers against each. Log which model actually served the request via the response model field.
Priority tiers
Mark internal eval traffic as low priority so user requests preempt it when the unified bucket runs low. The gateway should expose a header like x-priority: high and shed low-priority load first.
5. Forward cache-control hints
Providers like Anthropic and OpenAI offer prompt caching with discounted token rates, but only if you send cache-control markers. Your gateway must forward these untouched, or you lose the discount and inflate token usage against your unified quota.
resp = client.chat.completions.create(
model="anthropic/claude-3-5-sonnet",
messages=[
{"role": "system", "content": "Long system prompt", "cache_control": {"type": "ephemeral"}},
{"role": "user", "content": "Query"},
],
)If the gateway strips unknown fields, caching silently disables. Verify by checking provider responses for cache_read_input_tokens in the usage block. A correct gateway forwards provider cache-control hints exactly as sent.
Tradeoff: cache locality
Cache hits are per-provider. Fallback invalidates the cache, so a flapping primary increases token cost even if quota holds. Stabilize primary routes before relying on caching economics.
6. Meter and alert on the unified bucket
Per-token metering is only useful if you alert before the bucket empties. Export token counts per API key to your metrics system every minute.
# example: query gateway usage API
curl -H "Authorization: Bearer $KEY" \
https://gateway.example.com/v1/usage?window=1m
# => {"api_key":"backend-svc","tpm_used":41200,"tpm_limit":50000}Common mistake: ignoring streaming token counts. A streaming completion emits tokens over time; the gateway should count them as they flush, not at request end. Otherwise a long stream can exceed quota unnoticed. Build a dashboard that shows tpm_used / tpm_limit per key and page at 80%.
Cost attribution
Unified quota is not unified cost. Different models charge different rates per token. Use the gateway’s per-token metering to attribute spend back to teams even when the quota is shared.
7. Load test the collapse
Simulate your peak token rate multiplied by 1.5 against the gateway with fallbacks enabled. Watch for thundering-herd fallback: if primary fails, all traffic shifts at once and may trip the secondary’s per-provider limit—except your unified quota should absorb it because the gateway maps both to the same token budget. Confirm the gateway’s provider-specific caps don’t leak through.
# locust snippet
from locust import HttpUser, task
class LLMUser(HttpUser):
@task
def complete(self):
self.client.post("/v1/chat/completions", json={
"model":"openai/gpt-4o",
"messages":[{"role":"user","content":"x"*2000}]
})Run the test with the primary provider artificially blocked (return 429 at the gateway level) to validate fallback latency and quota behavior. Record p99 completion time with fallback active; if it exceeds your SLA, narrow fallback candidates to faster models.
Tradeoffs and final caveats
Collapsing per-provider rate limits to unified quota reduces application complexity but centralizes failure. If the gateway goes down, all models are unreachable. Mitigate with a health-checked deployment and a static fallback to direct calls for critical paths only.
You also lose fine-grained provider-specific controls unless the gateway exposes them. Ensure it honors client routing directives and cache hints as discussed; otherwise you pay hidden token costs. Keep a thin direct-call path for provider-only features (fine-tuning APIs, batch endpoints) that the gateway does not yet proxy.
The end state: your app sends a model name and a prompt, and a single quota protects the whole fleet. That is the practical definition of treating LLMs as infrastructure.