Rate limiting LLM requests in Cloudflare Workers

Cloudflare Workers rate limiting LLM requests is non-negotiable once you put a generative API in front of unpredictable traffic. A misbehaving client can drain your token budget or trigger provider 429s in seconds, so you need a counter at the edge that rejects overflow before it reaches the model. This guide walks through a token-bucket implementation using Durable Objects that you can ship today.

Step 1: Pick the right rate limit algorithm

Fixed windows are trivial but allow burst doubling at boundaries: a client can send limit requests at 00:59 and another limit at 01:00. Token bucket smooths bursts and is the right default for LLM calls where a single request can cost thousands of tokens.

Implement token bucket in a Durable Object. KV works for coarse global limits but suffers from eventual consistency; Durable Objects give single-key serializability per limit subject, which is exactly what you need for per-key or per-IP accounting.

Subject selection

Use a stable identifier: authenticated API key hash, or fallback to request.headers.get('CF-Connecting-IP'). Never rate limit purely on IP if you support keys—a shared NAT will collapse distinct users into one bucket.

Step 2: Initialize the Worker and Durable Object

Create the project and add the binding:

npm create cloudflare@latest rate-limit-llm
cd rate-limit-llm

Edit wrangler.toml:

name = "rate-limit-llm"
main = "src/index.ts"
compatibility_date = "2024-09-23"

[[durable_objects.bindings]]
name = "RATE_LIMITER"
class_name = "RateLimiter"

[[migrations]]
tag = "v1"
new_classes = ["RateLimiter"]

Run wrangler deploy once to register the migration before using the object.

Step 3: Build the token bucket Durable Object

The object stores tokens and lastRefill, then refills based on elapsed time. Wrap storage reads/writes in blockConcurrencyWhile to avoid races when multiple requests hit the same key simultaneously.

export class RateLimiter {
  state: DurableObjectState;
  constructor(state: DurableObjectState) {
    this.state = state;
  }

  async fetch(request: Request): Promise<Response> {
    const url = new URL(request.url);
    const cost = Number(url.searchParams.get('cost') ?? '1');
    const limit = Number(url.searchParams.get('limit') ?? '10');
    const refillPerSec = Number(url.searchParams.get('refill') ?? '1');

    return await this.state.blockConcurrencyWhile(async () => {
      let data = await this.state.storage.get(['tokens', 'updated']) as any;
      if (!data) data = { tokens: limit, updated: Date.now() };
      const now = Date.now();
      const elapsed = (now - data.updated) / 1000;
      const tokens = Math.min(limit, data.tokens + elapsed * refillPerSec);

      if (tokens < cost) {
        const wait = Math.ceil((cost - tokens) / refillPerSec);
        return new Response(JSON.stringify({ error: 'rate_limited' }), {
          status: 429,
          headers: { 'Retry-After': String(wait) }
        });
      }
      const remaining = tokens - cost;
      await this.state.storage.put({ tokens: remaining, updated: now });
      return new Response(JSON.stringify({ ok: true, remaining }), { status: 200 });
    });
  }
}

This is minimal but correct. In production, validate cost, limit, and refill ranges to prevent a client from passing refill=100000.

Step 4: Enforce the limit in the Worker entrypoint

In src/index.ts, derive the subject and call the limiter before proxying. Use idFromName so each key maps to a dedicated object instance.

export default {
  async fetch(request: Request, env: { RATE_LIMITER: DurableObjectNamespace }): Promise<Response> {
    const key = request.headers.get('x-api-key') ?? request.headers.get('CF-Connecting-IP') ?? 'anon';
    const id = env.RATE_LIMITER.idFromName(key);
    const stub = env.RATE_LIMITER.get(id);

    const check = await stub.fetch(`https://limiter/?cost=1&limit=20&refill=0.33`);
    if (check.status === 429) return check;

    return proxyToLLM(request, env);
  }
};

The example allows 20 requests then refills at 0.33/sec (~20 per minute). Tune these to your provider quota and pricing tolerance.

Step 5: Proxy the LLM completion call

Forward the body to an OpenAI-compatible endpoint. Because n4n.ai honors client routing directives and forwards provider cache-control hints, pass any cache-control or routing headers verbatim. If you route through a gateway such as n4n.ai, you also get automatic fallback when a provider is rate-limited or degraded, but your cloudflare workers rate limiting llm logic still shields that gateway from abusive clients.

async function proxyToLLM(request: Request, env: Env): Promise<Response> {
  const apiBase = env.LLM_API_BASE ?? 'https://api.openai.com/v1';
  const apiKey = env.LLM_API_KEY;
  const body = await request.text();

  const headers: Record<string, string> = {
    'content-type': 'application/json',
    authorization: `Bearer ${apiKey}`
  };
  const cc = request.headers.get('cache-control');
  if (cc) headers['cache-control'] = cc;

  const upstream = await fetch(`${apiBase}/chat/completions`, {
    method: 'POST',
    headers,
    body
  });

  return new Response(upstream.body, {
    status: upstream.status,
    headers: upstream.headers
  });
}

Set secrets:

wrangler secret put LLM_API_KEY
wrangler secret put LLM_API_BASE

Step 6: Charge cost by estimated token count

A per-request limit is crude for LLMs. Improve it by estimating prompt tokens and using that as cost. A simple heuristic is four bytes per token for English; for production use a proper BPE tokenizer at the edge if you can fit one.

function estimateTokens(text: string): number {
  return Math.ceil(new TextEncoder().encode(text).length / 4);
}

// Inside fetch, before limiter call:
const reqBody = await request.clone().json().catch(() => ({}) as any);
const prompt = reqBody.messages?.map((m: any) => m.content).join(' ') ?? '';
const cost = estimateTokens(prompt);
const check = await stub.fetch(`https://limiter/?cost=${cost}&limit=2000&refill=33`);

This caps each key to ~2000 tokens with a 33 token/sec refill (roughly 100k/hour). Deduct cost before streaming starts; do not refund on success.

Step 7: Verify locally with wrangler dev

Run the Worker:

wrangler dev --local

Hammer the endpoint from another shell:

for i in $(seq 1 30); do
  curl -s -o /dev/null -w "%{http_code}\n" \
    -X POST http://localhost:8787 \
    -H "x-api-key: test" \
    -H "content-type: application/json" \
    -d '{"messages":[{"role":"user","content":"hello"}]}'
done

The first ~20 requests return 200; the rest return 429 with a Retry-After header. That confirms cloudflare workers rate limiting llm requests works at the edge. For token-cost mode, send a large prompt and inspect the remaining field in the limiter’s JSON response by hitting the stub directly or logging it.

Step 8: Deploy and observe

wrangler deploy

Watch Workers analytics for the 429 ratio. If legitimate traffic hits limits, raise limit or refill per tier. Durable Objects bill per request; a few million limit checks per month costs cents, not dollars.

For global low-latency, Durable Objects are global but pin to a region on first access. That is fine for per-key limits because a key’s requests are usually correlated to one geography.

Caveats and hard-won lessons

• Reject before streaming. Never start the upstream fetch and then decide to 429—you already spent tokens.
• If you use server-sent events, deduct cost up front. On upstream error, you may choose to refund, but only inside a blockConcurrencyWhile to keep the bucket consistent.
• CF-Connecting-IP is synthesized in wrangler dev --local; for exact edge behavior use wrangler dev --remote or deploy.
• For anonymous traffic, combine IP with a sliding window or require a lightweight signed token to prevent one office NAT from starving others.

Cloudflare workers rate limiting llm traffic is not glamorous, but it is the difference between a manageable bill and a panic page at 3am. Ship the Durable Object, test the 429 path, and move on.