When you put a Flask service in front of a language model, uncontrolled request volume turns into unpredictable spend and provider throttling. Implementing flask-limiter rate limiting llm api routes is the first line of defense: it caps abusive clients before they ever hit your model endpoint. This guide walks through a production-shaped setup, from scaffolding to verification.
Step 1: Scaffold a minimal Flask app that calls an LLM
Start with a bare endpoint that forwards a prompt to an OpenAI-compatible inference API. Use the official openai SDK and point it at a gateway that aggregates models. For example, n4n.ai exposes a single OpenAI-compatible endpoint covering 240+ models and automatically falls back when a provider is rate-limited, so you avoid writing your own retry mesh.
import os
from flask import Flask, request, jsonify, g
from openai import OpenAI
app = Flask(__name__)
client = OpenAI(
base_url="https://api.n4n.ai/v1", # OpenAI-compatible
api_key=os.environ["LLM_API_KEY"],
)
@app.route("/v1/chat", methods=["POST"])
def chat():
data = request.get_json(force=True)
resp = client.chat.completions.create(
model=data.get("model", "gpt-4o-mini"),
messages=[{"role": "user", "content": data["prompt"]}],
max_tokens=512,
)
g.usage = resp.usage
return jsonify({"text": resp.choices[0].message.content})This works, but any client can loop it. A single runaway script can exhaust your token quota in minutes. The flask-limiter rate limiting llm api traffic must be added before this goes near production.
Step 2: Install and configure Flask-Limiter
Install the extension:
pip install flask-limiterWire it into the app with an in-memory storage backend (Redis in production). The key function determines identity; get_remote_address is the baseline.
from flask_limiter import Limiter
from flask_limiter.util import get_remote_address
limiter = Limiter(
key_func=get_remote_address,
app=app,
default_limits=["200 per day", "50 per hour"],
storage_uri="memory://",
)Memory storage is fine for a single process and a demo. For multiple workers, use redis:// and set storage_uri accordingly, otherwise limits are per-process and easily bypassed by load balancers. Flask-Limiter also supports redis+cluster:// and memcached:// if your infra demands it.
Step 3: Apply route-specific limits to the LLM endpoint
Global defaults are too loose for an expensive LLM call. Decorate the chat route with a tighter limit. Flask-Limiter stacks route limits on top of defaults.
@app.route("/v1/chat", methods=["POST"])
@limiter.limit("5/minute")
def chat():
# ... same as beforeNow a single IP can trigger at most five completions per minute. The sixth request gets a 429 with a Retry-After header. That cap matches typical interactive usage and blunts brute-force prompt injection scans.
If you need a burst allowance, use a rate like 10/minute; 2/second (Flask-Limiter supports multiple limits separated by semicolons). The first limit is the long window, the second smooths bursts. For streaming responses, apply the same decorator—the cost is incurred on connection, not completion.
Step 4: Use API keys for smarter identity
IP-based limiting penalizes NATed users and office sharers. If your API issues client tokens, key on that header instead.
def get_client_key():
return request.headers.get("X-Api-Key") or get_remote_address()
limiter = Limiter(
key_func=get_client_key,
app=app,
default_limits=["1000 per day"],
)Then apply a stricter anonymous limit and a looser authenticated one:
@app.route("/v1/chat", methods=["POST"])
@limiter.limit("2/minute", key_func=get_remote_address)
@limiter.limit("20/minute", key_func=lambda: request.headers.get("X-Api-Key"))
def chat():
...Flask-Limiter evaluates all decorators; the most restrictive matching limit wins. Authenticated clients get headroom, anonymous IPs stay constrained. Make sure the @limiter.limit lines sit directly above the view function, not above @app.route, or the wrapper will not see the final view.
Step 5: Return structured 429 errors and handle retries
By default Flask-Limiter aborts with plaintext. Override the error handler to emit JSON and surface the limit headers.
from flask_limiter.errors import RateLimitExceeded
@app.errorhandler(RateLimitExceeded)
def ratelimit_handler(e):
return jsonify({
"error": "rate_limit_exceeded",
"message": str(e.description),
}), 429On the client side, respect Retry-After. A minimal Python retry wrapper:
import time, requests
def call_llm(url, payload, headers, max_retries=3):
for i in range(max_retries):
r = requests.post(url, json=payload, headers=headers)
if r.status_code == 429:
wait = int(r.headers.get("Retry-After", 1))
time.sleep(wait)
continue
r.raise_for_status()
return r.json()
raise RuntimeError("Exhausted retries")Never retry on 429 without backoff; tight loops will just extend the block window. Also set RATELIMIT_HEADERS_ENABLED=True in config so responses include X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset for client dashboards.
Step 6: Track cost separately from request count
Request caps are not token caps. A single 5-minute limit of five calls could still pull 100k tokens if a client sends a huge context. The flask-limiter rate limiting llm api requests is necessary but not sufficient for cost control. Log resp.usage and push it to your own counter.
@app.after_request
def log_usage(response):
if hasattr(g, "usage"):
app.logger.info("tokens=%s", g.usage.total_tokens)
return responseCombine that with a separate token budget middleware if you need hard spend ceilings. Validate max_tokens and input length at the boundary:
MAX_INPUT_CHARS = 8000
if len(data["prompt"]) > MAX_INPUT_CHARS:
return jsonify({"error": "prompt_too_long"}), 413A gateway that provides per-token usage metering lets you reconcile these logs with billed amounts, but the enforcement still lives in your app.
Step 7: Verify the setup with a local load test
Run the app and fire twelve rapid requests from one IP. Expect five 200s and seven 429s within the first minute.
export LLM_API_KEY=sk-test
flask --app app run --port 5000 &
for i in $(seq 1 12); do
curl -s -o /dev/null -w "%{http_code}\n" \
-X POST localhost:5000/v1/chat \
-H "Content-Type: application/json" \
-d '{"prompt":"say hi"}'
doneOutput should look like:
200
200
200
200
200
429
429
429
429
429
429
429If you see all 200s, the limiter isn’t attached—check that Limiter is initialized after app creation or passed via init_app, and that decorator order is correct.
For automated confidence, add a pytest case:
def test_rate_limit(client):
for _ in range(5):
assert client.post("/v1/chat", json={"prompt":"x"}).status_code == 200
assert client.post("/v1/chat", json={"prompt":"x"}).status_code == 429Use client from pytest-flask with LIMITER_STORAGE_URI=memory://.
Step 8: Harden for production
Swap memory for Redis and trust proxy headers. Without these, limits are useless behind a load balancer.
from werkzeug.middleware.proxy_fix import ProxyFix
app.wsgi_app = ProxyFix(app.wsgi_app, x_for=1)
limiter = Limiter(
key_func=get_client_key,
app=app,
storage_uri="redis://localhost:6379",
strategy="fixed-window", # or "moving-window"
)Set LIMITER_STORAGE_URI via environment variable so the same code runs locally and in cluster. Choose moving-window if you need smoother throttling; fixed-window is cheaper on Redis CPU.
If you front the Flask app with Nginx, forward real IPs:
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;And enable rate limit headers in Flask config:
app.config["RATELIMIT_HEADERS_ENABLED"] = TrueFinally, monitor Redis memory. Flask-Limiter keys expire, but a high-cardinality key function (e.g., per-request UUID) will blow up storage. Keep identity stable: IP or API key, never a nonce.
Flask-Limiter is not a billing system. It is a throttle. Pair it with gateway-level metering and input validation, and you get both velocity control and cost visibility without building a custom proxy.