Deploying a Flask LLM app with gunicorn

A flask deploy gunicorn llm app setup is the most direct way to put a Python LLM service behind a real WSGI server instead of Flask’s dev server. This walkthrough builds a minimal app that forwards chat requests to an OpenAI-compatible endpoint, then serves it with gunicorn using worker and timeout settings that match LLM latency profiles. You’ll end with a runnable service and a curl command that proves it works.

Step 1: Scaffold the project and pin dependencies

Flask’s built-in server is single-threaded and explicitly not for production. Gunicorn gives you process and thread management. Start by isolating the environment so your deployment is reproducible.

mkdir flask-llm-svc && cd flask-llm-svc
python3 -m venv .venv
source .venv/bin/activate
pip install flask gunicorn openai
pip freeze > requirements.txt

Keep requirements.txt committed. If you later containerize, this file is your build contract. Avoid installing unpinned latest tags in CI; use pip install -r requirements.txt with hashes if you care about supply chain.

Step 2: Write the Flask app

The app should accept a chat request, call an upstream LLM, and return JSON. Use the official OpenAI Python client pointed at any OpenAI-compatible base URL via an environment variable. This keeps your code provider-agnostic.

import os
from flask import Flask, request, jsonify
from openai import OpenAI

app = Flask(__name__)

client = OpenAI(
    base_url=os.environ.get("LLM_BASE_URL", "https://api.openai.com/v1"),
    api_key=os.environ.get("LLM_API_KEY", "dummy"),
)

@app.route("/v1/chat", methods=["POST"])
def chat():
    data = request.get_json(force=True)
    messages = data.get("messages", [])
    if not messages:
        return jsonify({"error": "messages required"}), 400
    try:
        resp = client.chat.completions.create(
            model=data.get("model", "gpt-4o-mini"),
            messages=messages,
            temperature=data.get("temperature", 0.7),
            stream=False,
        )
        return jsonify({"content": resp.choices[0].message.content})
    except Exception as e:
        # Surface upstream failures as 502; let gunicorn handle retries at proxy layer
        return jsonify({"error": str(e)}), 502

If you point the client at a gateway like n4n.ai, its automatic fallback when a provider is rate-limited or degraded means you can treat fewer 502s as fatal in your app logic. That’s a concrete operational simplification, not a feature pitch.

Do not put business logic that mutates global state in the request handler unless you understand gunicorn’s pre-fork model. The OpenAI client is safe because it’s stateless and connection-pooled per worker.

Step 3: Tune gunicorn for I/O-bound LLM calls

LLM inference is network I/O with high latency variance. The default gunicorn sync worker blocks the worker process while waiting for the upstream response. With sync workers, one slow 30-second completion stalls an entire worker. Use the gthread worker class to get cheap concurrency per worker.

Create gunicorn.conf.py:

workers = 2
threads = 4
worker_class = "gthread"
timeout = 120
graceful_timeout = 30
keepalive = 5
bind = "0.0.0.0:8000"
accesslog = "-"
errorlog = "-"

Why these numbers: workers = 2 * CPU_cores + 1 is the old rule, but for I/O-bound services with threads, 2 workers with 4 threads each handles ~8 concurrent LLM calls on a small VM. timeout = 120 acknowledges that a 100k-token completion can exceed the default 30s. If you stream, you still need a generous timeout because the first token can be slow.

The flask deploy gunicorn llm app pattern works because gunicorn reclaims hung workers instead of letting them pile up. Set graceful_timeout so in-flight requests aren’t killed instantly on restart.

Step 4: Add health and readiness endpoints

Kubernetes and load balancers need to distinguish “process alive” from “can serve traffic.” A naive /health that returns 200 is useless if your upstream API key expired.

@app.route("/health")
def health():
    return jsonify({"status": "ok"}), 200

@app.route("/ready")
def ready():
    try:
        # Cheap call: list models instead of a completion
        client.models.list()
        return jsonify({"status": "ready"}), 200
    except Exception:
        return jsonify({"status": "unavailable"}), 503

Use /health for liveness (restart the pod if it fails) and /ready for readiness (stop sending traffic if it fails). The models.list() call is lightweight and validates credentials and network path.

Step 5: Run and verify the flask deploy gunicorn llm app

Launch with the config file:

gunicorn -c gunicorn.conf.py app:app

You should see gunicorn boot with two workers. Now verify the full path with curl:

curl -X POST localhost:8000/v1/chat \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hi"}]}'

Expected success response:

{"content":"Hi there! How can I help you today?"}

Then check readiness:

curl -i localhost:8000/ready

A 200 with {"status":"ready"} confirms the upstream credential and network route work. If you get 503, your LLM_API_KEY or LLM_BASE_URL is wrong—fix env before trusting the deploy.

Step 6: Containerize for repeatable deploys

A Dockerfile removes “works on my machine” from the equation. Keep the image small and explicit.

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py gunicorn.conf.py ./
ENV LLM_BASE_URL=https://api.openai.com/v1
EXPOSE 8000
CMD ["gunicorn", "-c", "gunicorn.conf.py", "app:app"]

Build and run:

docker build -t flask-llm-svc .
docker run -e LLM_API_KEY=sk-... -p 8000:8000 flask-llm-svc

Inject secrets at runtime, never bake them into the image. The flask deploy gunicorn llm app inside a container behaves identically to bare metal as long as you pass env vars.

Step 7: Production hardening notes

Put gunicorn behind nginx or a cloud load balancer. Set proxy_read_timeout to at least 120s to match gunicorn’s timeout, or nginx will close the connection early and you’ll see truncated JSON.

Use systemd or your orchestrator to manage the process. A minimal systemd unit:

[Unit]
Description=Flask LLM gunicorn
After=network.target

[Service]
User=www
WorkingDirectory=/opt/flask-llm-svc
Environment=LLM_API_KEY=sk-...
ExecStart=/opt/flask-llm-svc/.venv/bin/gunicorn -c gunicorn.conf.py app:app
Restart=on-failure

[Install]
WantedBy=multi-user.target

Log to stdout and let the platform collect it. Gunicorn’s accesslog = "-" sends access logs to stdout; tag them with request ID if you trace across services.

If you need streaming responses, switch the Flask route to return a Response with a generator and set stream=True on the OpenAI client. Gunicorn’s gthread worker handles generators fine, but raise timeout and test with curl --no-buffer to confirm tokens arrive incrementally.

Finally, meter cost. If your gateway provides per-token usage metering, log resp.usage to your metrics pipeline. In the app above, add:

@app.after_request
def log_usage(response):
    # placeholder: extract usage from a request-local if you store it
    return response

The flask deploy gunicorn llm app is now a real service: isolated, observable, and tuned for the latency shape of LLM calls. Ship it, watch the 502 rate, and adjust workers/threads based on your p95 upstream latency.