How to add rate limiting to a Django LLM API

Building a public-facing django rate limiting llm api is non-negotiable once you put a paid model behind it. Without throttling, a single script or a misconfigured client can exhaust your token budget in minutes and leave every other user with 429s.

Step 1: Create the Django project and LLM proxy view

Start with a minimal DRF app that forwards prompts to an OpenAI-compatible endpoint. Install the dependencies:

pip install django djangorestframework openai redis

Define a view that accepts a prompt and returns the model completion:

# views.py
import os
from rest_framework.views import APIView
from rest_framework.response import Response
from openai import OpenAI

client = OpenAI(
    base_url="https://api.openai.com/v1",
    api_key=os.environ["OPENAI_API_KEY"],
)

class ChatView(APIView):
    def post(self, request):
        prompt = request.data.get("prompt", "")
        if not prompt:
            return Response({"error": "prompt required"}, status=400)
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
        )
        return Response({"text": resp.choices[0].message.content})

Wire it in urls.py:

from django.urls import path
from .views import ChatView

urlpatterns = [path("chat", ChatView.as_view())]

This works, but it has zero protection. Anyone can loop it.

Step 2: Decide between request-count and token-count throttling

A naive request limiter (e.g., 100 requests/hour) is easy but wrong for LLMs. One request with a 10k-token document costs far more than 100 short requests. For a django rate limiting llm api, you must throttle on estimated token consumption, not just HTTP calls.

Token estimation does not need to be exact. A rough heuristic—len(text) / 4 for English—is enough for throttling purposes. Pair that with a refill rate per hour to cap spend.

Step 3: Stand up Redis for shared throttle state

Django runs multiple workers. In-memory counters will drift. Use Redis as the shared store:

docker run -d -p 6379:6379 redis:7

Point Django at it via settings.py:

REDIS_HOST = "localhost"
REDIS_PORT = 6379
REDIS_DB = 0

Step 4: Add a basic request-rate throttle with DRF

Before the token bucket, drop in DRF’s built-in UserRateThrottle to block obvious abuse:

# settings.py
REST_FRAMEWORK = {
    "DEFAULT_THROTTLE_CLASSES": ["rest_framework.throttling.UserRateThrottle"],
    "DEFAULT_THROTTLE_RATES": {"user": "60/minute"},
}

This caps every authenticated user to 60 requests/minute regardless of payload size. It is a first line of defense, not a cost control.

Step 5: Replace it with a token-bucket throttle for LLM spend

Implement a custom throttle that deducts estimated tokens from a per-user bucket that refills over time:

# throttles.py
import time
import redis
from rest_framework.throttling import BaseThrottle
from django.conf import settings

r = redis.Redis(
    host=settings.REDIS_HOST,
    port=settings.REDIS_PORT,
    db=settings.REDIS_DB,
)

class TokenBucketThrottle(BaseThrottle):
    BURST_TOKENS = 20_000
    REFILL_PER_SEC = 50_000 / 3600  # ~50k tokens/hour
    PREFIX = "llm_bucket:"

    def get_key(self, request):
        if request.user.is_authenticated:
            return f"{self.PREFIX}{request.user.id}"
        return f"{self.PREFIX}{request.META['REMOTE_ADDR']}"

    def allow_request(self, request, view):
        key = self.get_key(request)
        now = time.time()
        est = max(1, int(len(request.data.get("prompt", "")) / 4))

        data = r.hgetall(key)
        if not data:
            tokens = self.BURST_TOKENS
            ts = now
        else:
            tokens = float(data[b"tokens"])
            ts = float(data[b"ts"])

        tokens = min(self.BURST_TOKENS, tokens + (now - ts) * self.REFILL_PER_SEC)
        if tokens < est:
            return False

        r.hset(key, mapping={"tokens": tokens - est, "ts": now})
        r.expire(key, 3600)
        return True

    def wait(self, request, view):
        return None

Apply it to the view:

from .throttles import TokenBucketThrottle

class ChatView(APIView):
    throttle_classes = [TokenBucketThrottle]
    # ...

Now a user who dumps a 16k-token prompt burns most of their hourly quota in one call.

Step 6: Apply per-user and per-model scopes

Hard-coding limits in the class is fine for a demo. In production, load quotas from your user table:

def get_quota(self, request):
    if request.user.is_authenticated:
        return request.user.profile.monthly_token_quota
    return 10_000  # anon ceiling

For multi-model routing, add a scope keyed on request.data.get("model"). DRF’s ScopedRateThrottle can do request-rate scoping; for token scoping, extend the Redis key with the model name. This prevents a cheap model from subsidizing a flagship one.

Step 7: Deal with upstream provider rate limits

Your throttle protects your wallet, but the upstream LLM provider can still return 429s under load. If you front your calls with a gateway such as n4n.ai, its automatic fallback when a provider is rate-limited or degraded reduces the chance of hard failures, but client-side django rate limiting llm api throttling remains the only way to cap spend per tenant. Always catch openai.RateLimitError and return a structured 429 with Retry-After.

from openai import RateLimitError
try:
    resp = client.chat.completions.create(...)
except RateLimitError:
    return Response({"error": "upstream limit"}, status=429)

Step 8: Verify the django rate limiting llm api end to end

Run the dev server and hammer the endpoint with a loop:

for i in {1..10}; do
  curl -s -X POST localhost:8000/chat \
    -H 'Content-Type: application/json' \
    -d '{"prompt":"Explain distributed systems in detail."}' \
    -w "\n%{http_code}\n"
done

You should see 200 for the first few calls, then 429 once the bucket drains. Inspect Redis to confirm state:

redis-cli HGETALL llm_bucket:1

You will see tokens drop and ts update. Reset with redis-cli DEL llm_bucket:1 to re-test.

Production considerations

• Run the throttle check before parsing the body if you want to reject early; DRF parses JSON before allow_request, so use a lightweight middleware if payload size is a concern.
• Log denied requests with the estimated token count. That data tells you whether your refill rate is realistic.
• For async views, use redis.asyncio and async def allow_request.
• Per-token metering at the gateway (e.g., per-token usage metering from your provider) should reconcile with your throttle numbers nightly. Discrepancies mean your estimator is off.

A django rate limiting llm api is not a luxury. It is the difference between a demo and a bill you can predict. Ship the token bucket before you ship the model.