Deploying a FastAPI LLM backend with uvicorn and gunicorn

A fastapi deploy uvicorn gunicorn llm backend is the default pattern for shipping Python inference APIs that need concurrency without rewriting your model client. You get ASGI request handling from FastAPI, a lightweight server loop from uvicorn, and process isolation from gunicorn. This guide walks through a production-shaped setup you can copy into a repo today.

Step 1: Scaffold the project and pin dependencies

Create a clean virtual environment and install the minimum set of packages. Avoid unpinned installs in production—gunicorn and uvicorn have breaking changes across major versions.

mkdir llm-gateway && cd llm-gateway
python -m venv .venv && source .venv/bin/activate
pip install "fastapi==0.111.0" "uvicorn[standard]==0.30.1" "gunicorn==22.0.0" "openai==1.35.0" "pydantic==2.7.1"

Write a requirements.txt from the locked versions so your container build is reproducible:

fastapi==0.111.0
uvicorn[standard]==0.30.1
gunicorn==22.0.0
openai==1.35.0
pydantic==2.7.1

A fastapi deploy uvicorn gunicorn llm service should keep its config explicit. Put environment-specific values (API keys, model name, base URL) in a .env file loaded at startup, not hardcoded.

Step 2: Build the FastAPI app with async LLM routes

The core mistake engineers make is blocking the event loop with synchronous SDK calls. Use the async client. Below is a minimal but realistic /v1/chat endpoint that proxies to an OpenAI-compatible backend.

# main.py
import os
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from openai import AsyncOpenAI

app = FastAPI(title="llm-backend")

client = AsyncOpenAI(
    api_key=os.environ["LLM_API_KEY"],
    base_url=os.environ.get("LLM_BASE_URL", "https://api.openai.com/v1"),
)

class ChatRequest(BaseModel):
    model: str = "gpt-4o-mini"
    messages: list[dict]
    stream: bool = False

@app.post("/v1/chat")
async def chat(req: ChatRequest):
    try:
        if req.stream:
            # Return SSE manually or use StreamingResponse; omitted for brevity
            raise HTTPException(status_code=501, detail="stream not shown")
        resp = await client.chat.completions.create(
            model=req.model,
            messages=req.messages,
            stream=False,
        )
        return {"content": resp.choices[0].message.content, "usage": resp.usage.model_dump()}
    except Exception as e:
        raise HTTPException(status_code=502, detail=str(e))

@app.get("/health")
async def health():
    return {"status": "ok"}

If you want to avoid juggling multiple provider keys, point LLM_BASE_URL at a gateway that exposes one OpenAI-compatible endpoint for 240+ models and automatically falls back when a provider is rate-limited. The FastAPI code above does not change.

Step 3: Run uvicorn standalone for local development

Before involving gunicorn, confirm the app boots under uvicorn with the standard worker. Use --reload only locally.

uvicorn main:app --host 0.0.0.0 --port 8000 --reload

Hit the health check:

curl localhost:8000/health
# {"status":"ok"}

For a fastapi deploy uvicorn gunicorn llm stack, uvicorn alone is fine for a single process, but it does not manage multiple worker processes or graceful restarts. That is gunicorn’s job.

Step 4: Configure gunicorn with UvicornWorker

Gunicorn is a WSGI server, but with the UvicornWorker class it delegates ASGI handling to uvicorn inside each forked process. Create gunicorn.conf.py:

# gunicorn.conf.py
import multiprocessing

bind = "0.0.0.0:8000"
workers = multiprocessing.cpu_count() * 2 + 1
worker_class = "uvicorn.workers.UvicornWorker"
timeout = 120          # LLM calls can be slow; default 30s is too tight
graceful_timeout = 30
keepalive = 5
accesslog = "-"
errorlog = "-"
loglevel = "info"

For IO-bound LLM backends, you can push workers higher than the CPU-based formula because most time is spent awaiting network responses, not burning CPU. Monitor RSS per worker; a worker holding a large local model would need different math.

Launch:

gunicorn -c gunicorn.conf.py main:app

This is the canonical fastapi deploy uvicorn gunicorn llm command for production. Gunicorn forks N processes, each running an async event loop via uvicorn.

Step 5: Add a readiness probe and request limits

Kubernetes or systemd need a readiness signal beyond /health. Extend the app with a lightweight model warm-up check, and cap request size to avoid abuse.

from fastapi import Request
from fastapi.responses import JSONResponse

@app.middleware("http")
async def limit_body(request: Request, call_next):
    if request.headers.get("content-length") and int(request.headers["content-length"]) > 1_000_000:
        return JSONResponse(status_code=413, content={"error": "payload too large"})
    return await call_next(request)

Also set client_max_size in uvicorn if you serve file uploads, but for chat JSON the middleware suffices.

Step 6: Containerize the service

A Dockerfile makes the fastapi deploy uvicorn gunicorn llm artifact portable. Use a slim base and run as non-root.

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py gunicorn.conf.py ./
ENV PYTHONUNBUFFERED=1
EXPOSE 8000
USER nobody
CMD ["gunicorn", "-c", "gunicorn.conf.py", "main:app"]

Build and run:

docker build -t llm-backend .
docker run -e LLM_API_KEY=sk-... -p 8000:8000 llm-backend

If you need horizontal scale, place this container behind a load balancer. Gunicorn’s multiple workers already saturate a single node’s cores for IO-bound traffic.

Step 7: Verify end-to-end and under load

Success means: health check passes, a chat request returns tokens, and the process survives concurrent traffic.

Single request:

curl -X POST localhost:8000/v1/chat \
  -H "content-type: application/json" \
  -d '{"messages":[{"role":"user","content":"say hi"}]}'

Expected: JSON with content and usage fields, HTTP 200.

Concurrent load test with hey:

hey -z 30s -q 10 -m POST -H "content-type: application/json" \
  -d '{"messages":[{"role":"user","content":"ping"}]}' \
  http://localhost:8000/v1/chat

Watch gunicorn’s access log for 5xx rates. A well-tuned fastapi deploy uvicorn gunicorn llm backend should hold p99 latency near your model provider’s latency plus a few milliseconds of proxy overhead. If workers spike in memory, lower workers or switch to a streaming response to free connections earlier.

Step 8: Graceful shutdown and logging

Gunicorn forwards SIGTERM to workers; uvicorn shuts down the loop. Ensure your LLM client does not block shutdown—the async OpenAI client cancels in-flight requests cleanly. Set loglevel="info" and ship logs to stdout for container collectors.

One last note: never run uvicorn with --reload in production, and never expose the gunicorn stats socket publicly. The pattern above gives you a defensible baseline; from here, add auth middleware, per-tenant rate limits, and tracing.