Most LLM backends collapse under abusive traffic because token generation is expensive and slow. Adding express.js rate limiting express-rate-limit to your Express service is the fastest way to protect upstream model providers without building a custom proxy.
We’ll stand up a minimal Express server that proxies to an OpenAI-compatible endpoint, layer rate limits on it, and prove the limits hold under load.
Step 1: Scaffold a minimal Express LLM proxy
Start with a clean Node project. Use Node 18+ so you get global fetch.
mkdir llm-gateway-proxy && cd llm-gateway-proxy
npm init -y
npm install expressCreate server.js with a single POST route that forwards a chat completion request. Keep it dumb: validate nothing, just pass through.
import express from 'express';
import { env } from 'node:process';
const app = express();
app.use(express.json());
const UPSTREAM = env.UPSTREAM_URL || 'https://api.openai.com/v1/chat/completions';
const UPSTREAM_KEY = env.UPSTREAM_KEY;
app.post('/v1/chat/completions', async (req, res) => {
const upstreamRes = await fetch(UPSTREAM, {
method: 'POST',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${UPSTREAM_KEY}`,
},
body: JSON.stringify(req.body),
});
const data = await upstreamRes.json();
res.status(upstreamRes.status).json(data);
});
app.listen(3000, () => console.log('proxy on :3000'));This is enough to expose a working LLM endpoint. It will also explode the moment someone loops requests.
Step 2: Install and configure express-rate-limit
Install the middleware:
npm install express-rate-limitThe default in-memory store is fine for a single process. For multi-instance deployments you’ll swap in Redis later; we’ll cover that in Step 7.
Create a base limiter that caps each client to 30 requests per minute. That number is deliberately low for LLM endpoints—a single chat completion can burn thousands of tokens, and you don’t want a single rogue script to exhaust your provider quota.
import rateLimit from 'express-rate-limit';
const baseLimiter = rateLimit({
windowMs: 60_000,
max: 30,
standardHeaders: true, // send RateLimit-* headers
legacyHeaders: false, // omit X-RateLimit-* (deprecated)
message: { error: 'too many requests, slow down' },
});
app.use(baseLimiter);Apply it globally first to confirm behavior, then narrow scope in Step 3. With standardHeaders: true, clients receive RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset, plus Retry-After on 429s. Browsers and OpenAI SDKs understand these natively.
The sliding window used by express-rate-limit is a fixed counter per bucket, not a token bucket. That means a client can burst to max exactly at the window edge and again immediately after reset. For LLM traffic that’s acceptable—you’re protecting against sustained abuse, not shaving latency.
Step 3: Apply scoped limiters per route and identity
A global limiter treats your health check and your expensive completion route identically. Split them.
const chatLimiter = rateLimit({
windowMs: 60_000,
max: 10, // stricter: 10 completions/min per key
standardHeaders: true,
legacyHeaders: false,
keyGenerator: (req) => req.headers['x-api-key'] || req.ip,
handler: (req, res) => {
res.status(429).json({ error: 'rate limit exceeded for chat endpoint' });
},
});
const healthLimiter = rateLimit({
windowMs: 60_000,
max: 100,
keyGenerator: (req) => req.ip,
});
app.get('/health', healthLimiter, (req, res) => res.json({ ok: true }));
app.post('/v1/chat/completions', chatLimiter, async (req, res) => {
// same proxy code as Step 1
});Trust proxy and real client IP
If your Express app runs behind Nginx, ELB, or Cloudflare, req.ip resolves to the proxy address unless you set:
app.set('trust proxy', 1); // trust one hopWithout this, every client shares one limiter bucket and you’ll either block everyone or no one. Keying on x-api-key sidesteps the issue entirely, which is why we use it as the primary key above.
Dynamic limits by model
Heavy models deserve tighter caps. max accepts a function:
max: (req) => {
const model = req.body?.model || '';
if (model.includes('gpt-4') || model.includes('claude-opus')) return 5;
return 10;
}Because express.json() runs before the route handler, req.body is already parsed when the limiter evaluates.
Step 4: Proxy to a resilient LLM gateway
When you forward to a gateway that already handles provider failover, your Express layer only needs client-side throttling. For example, n4n.ai exposes one OpenAI-compatible endpoint across 240+ models and performs automatic fallback when a provider is rate-limited or degraded. Your code doesn’t change—just point UPSTREAM_URL at it and pass your gateway key.
const UPSTREAM = env.UPSTREAM_URL || 'https://api.n4n.ai/v1/chat/completions';Streaming responses need different handling. If you support stream: true, pipe the upstream body instead of buffering:
app.post('/v1/chat/completions', chatLimiter, async (req, res) => {
const upstreamRes = await fetch(UPSTREAM, {
method: 'POST',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${UPSTREAM_KEY}`,
},
body: JSON.stringify(req.body),
});
if (!upstreamRes.ok) {
const err = await upstreamRes.json();
return res.status(upstreamRes.status).json(err);
}
res.setHeader('content-type', 'text/event-stream');
upstreamRes.body.pipe(res);
});The rate limiter still applies because it intercepts the request before the upstream call. That’s the whole point: reject locally before spending a single token upstream.
Step 5: Return proper 429 responses and headers
express-rate-limit sends a 429 automatically when max is hit, but the default HTML page is useless for JSON APIs. We already set a custom handler in Step 3. Ensure you also surface the Retry-After header (set automatically with standardHeaders).
A robust pattern: catch upstream errors separately so a provider outage doesn’t get masked as a rate limit.
handler: (req, res, next) => {
const retry = res.getHeader('retry-after') || '60';
res.setHeader('retry-after', retry);
res.status(429).json({
error: 'rate_limited',
message: 'client exceeded request quota',
retry_after: Number(retry),
});
},Never return the upstream provider’s raw error to the client. It leaks your backend topology and confuses SDKs that expect OpenAI-shaped payloads.
Step 6: Load test with autocannon
Install a lightweight HTTP benchmark tool:
npm install -g autocannonHit the chat endpoint with 50 concurrent connections for 10 seconds, sending a tiny payload:
autocannon -c 50 -d 10 -m POST \
-H 'content-type: application/json' \
-H 'x-api-key: test-key' \
-b '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"hi"}]}' \
http://localhost:3000/v1/chat/completionsWith max: 10 per minute, autocannon will report a mix of 200 and 429 statuses. The throughput line should show roughly 10 successful requests per minute per key, not 500. If you see 500 successes, your limiter isn’t applied—check middleware order.
If you want programmatic verification in CI:
import autocannon from 'autocannon';
const result = await autocannon({
url: 'http://localhost:3000/v1/chat/completions',
method: 'POST',
headers: { 'x-api-key': 'test-key', 'content-type': 'application/json' },
body: JSON.stringify({ model: 'gpt-3.5-turbo', messages: [] }),
connections: 20,
duration: 5,
});
console.log(`2xx: ${result['2xx']}, 429: ${result['4xx']}`);
// expect 4xx > 0 and 2xx <= 10Step 7: Verify success and operationalize
Success means three things: legitimate single-user traffic completes, bursts get 429 with correct headers, and a second API key is throttled independently.
Check your server logs for the limiter’s warning (it logs only on misconfiguration). Instead, assert on response codes:
curl -i -X POST http://localhost:3000/v1/chat/completions \
-H 'x-api-key: key-a' -d '{"model":"x","messages":[]}'
# repeat 11 times within a minute -> 11th returns 429 with retry-afterFor production with multiple Express instances, the in-memory store breaks: each node counts separately, effectively multiplying limits by the number of pods. Swap in Redis:
npm install rate-limit-redis ioredisimport { RedisStore } from 'rate-limit-redis';
import Redis from 'ioredis';
const redisClient = new Redis(env.REDIS_URL);
const chatLimiter = rateLimit({
store: new RedisStore({
sendCommand: (...args) => redisClient.call(...args),
}),
windowMs: 60_000,
max: 10,
keyGenerator: (req) => req.headers['x-api-key'] || req.ip,
});Now limits are shared across the fleet. Combine this with per-token metering at the gateway (n4n.ai returns usage on each response) to build a two-layer defense: request caps at the edge, token caps at the billing layer.
Tune max based on real traffic. Pull your p95 request latency and upstream token cost; if a completion averages 2k output tokens and your provider quota is 200k tokens/min, 10 requests/min per key is already near saturation. Express.js rate limiting express-rate-limit is request-oriented, so pair it with upstream quotas rather than treating it as a token accountant.
Finally, version your limits. Put the numbers in env vars so you can adjust without redeploying code:
max: Number(env.CHAT_MAX_PER_MIN || 10),Add a tiny metrics middleware to count 429s in Prometheus or just console:
app.use((req, res, next) => {
res.on('finish', () => {
if (res.statusCode === 429) console.count('rate_limited');
});
next();
});That’s the whole path from a naive proxy to a throttled, multi-instance LLM API that survives a bored script kiddie.