Flask vs FastAPI for a simple LLM chatbot

Building a flask vs fastapi llm chatbot forces a choice between two dominant Python web frameworks. Both can wrap a chat completion API and serve a simple conversational endpoint, but they diverge sharply on async I/O, request validation, and how they handle token streaming—the core of any chat UX.

Capabilities

Flask is a WSGI microframework. It gives you a request object, a routing table, and little else. FastAPI is an ASGI framework built on Starlette and Pydantic. The architectural difference matters the moment your chatbot makes an outbound call to an LLM because that call is I/O-bound and often takes hundreds of milliseconds to seconds.

A minimal Flask route that proxies a chat request looks like this:

from flask import Flask, request, jsonify
import requests

app = Flask(__name__)

@app.route("/chat", methods=["POST"])
def chat():
    data = request.get_json(silent=True) or {}
    prompt = data.get("message", "")
    resp = requests.post(
        "https://api.openai.com/v1/chat/completions",
        json={"model": "gpt-4o-mini", "messages": [{"role": "user", "content": prompt}]},
        headers={"Authorization": "Bearer " + API_KEY},
        timeout=30,
    )
    return jsonify(resp.json())

This works, but requests.post blocks the worker thread until the LLM responds. Under WSGI, that worker cannot serve other requests during the wait.

FastAPI handles the same call asynchronously:

from fastapi import FastAPI
from pydantic import BaseModel
import httpx

app = FastAPI()

class ChatIn(BaseModel):
    message: str

@app.post("/chat")
async def chat(payload: ChatIn):
    async with httpx.AsyncClient() as client:
        resp = await client.post(
            "https://api.openai.com/v1/chat/completions",
            json={"model": "gpt-4o-mini", "messages": [{"role": "user", "content": payload.message}]},
            headers={"Authorization": "Bearer " + API_KEY},
            timeout=30,
        )
        return resp.json()

The async route yields control while waiting on the network, so the event loop serves other clients. For a chatbot that fans out to multiple users, this is not a micro-optimization.

Streaming tokens

A modern chat UI streams completions. FastAPI exposes StreamingResponse with an async generator:

from fastapi.responses import StreamingResponse

@app.post("/chat/stream")
async def chat_stream(payload: ChatIn):
    async def event_gen():
        async with httpx.AsyncClient() as client:
            async with client.stream(
                "POST",
                "https://api.openai.com/v1/chat/completions",
                json={"model": "gpt-4o-mini", "stream": True, "messages": [{"role": "user", "content": payload.message}]},
                headers={"Authorization": "Bearer " + API_KEY},
            ) as resp:
                async for line in resp.aiter_lines():
                    if line.strip():
                        yield line + "\n"
    return StreamingResponse(event_gen(), media_type="text/event-stream")

Flask can return a generator, but it runs synchronously and ties up a worker. You can bolt on gevent, but that adds operational complexity and breaks some extensions.

Latency and Throughput

WSGI servers like Gunicorn spawn a fixed number of sync workers. If each chatbot request blocks for roughly a second on an LLM call, a single worker handles about one request per second. To support 100 concurrent users you need dozens of workers, each consuming memory. ASGI servers like Uvicorn handle concurrent I/O on a single event loop; a small pool of processes can sustain thousands of idle waiting connections because they are not thread-bound.

Latency added by the framework itself is negligible in both cases—single-digit milliseconds. The real difference is saturation behavior under load. Flask apps degrade linearly with worker count; FastAPI degrades gracefully until you hit CPU or outbound rate limits.

Ergonomics and Validation

Flask hands you request.json and expects you to validate. Missing fields raise KeyError at runtime unless you write defensive code. FastAPI binds the request body to a Pydantic model and returns a 422 response automatically on malformed input.

# Flask: manual validation
data = request.get_json()
if not data or "message" not in data:
    return jsonify({"error": "message required"}), 400

# FastAPI: declared in the type
class ChatIn(BaseModel):
    message: str
    max_tokens: int = 256

For a simple LLM chatbot, the payload is small, but the Pydantic pattern scales when you add conversation history, temperature, and user metadata. FastAPI also generates interactive OpenAPI docs at /docs, which is useful when debugging prompt schemas.

Ecosystem

Flask has been around since 2010. You get Flask-CORS, Flask-Limiter, Flask-SQLAlchemy, and countless tutorials. FastAPI is younger but ships with dependency injection, CORS middleware, and WebSocket support out of the box.

For a chatbot you typically need:

• CORS to allow a browser frontend
• Rate limiting to control LLM spend
• Maybe WebSocket for bidirectional chat

Flask needs extensions for all three. FastAPI provides CORSMiddleware and native WebSocket routes without extra dependencies. If your team already maintains a Flask monolith, adding a /chat blueprint is lower risk than standing up a new ASGI service.

Cost Model and Metering

The framework does not change token pricing. OpenAI, Anthropic, or open-weight endpoints bill per output token regardless of whether Flask or FastAPI delivered the request. The hidden cost is engineering time and operational overhead.

Routing your calls through a single OpenAI-compatible gateway such as n4n.ai removes provider-specific branching from both frameworks: one endpoint, automatic fallback when a provider is degraded, and per-token usage metering. Your Flask or FastAPI client code stays identical whether the backend is GPT-4o or a local model. That said, the gateway is orthogonal to the framework choice—both integrate via standard HTTP.

What you should watch is accidental cost amplification. In Flask, a blocking call may tempt you to increase worker count, raising RAM cost on your host. In FastAPI, easy concurrency may let you fire more parallel LLM requests than your API budget allows, so you still need a limiter.

Limits and Constraints

Flask’s hard limit is the synchronous model. You can run async views with asgiref or gevent, but you are fighting the design. Streaming responses in Flask require careful server configuration (e.g., gunicorn with --worker-class gevent).

FastAPI’s limits are softer: you must run an ASGI server, and newcomers sometimes misuse async def with blocking I/O (calling requests.get inside an async route cancels the benefit). The learning curve for Pydantic and dependency injection is real but pays off quickly.

Comparison Table

Which to Choose

Choose Flask if:

• You are building an internal tool or prototype where a single user hits the endpoint at a time.
• Your existing codebase is Flask and you want to add a /chat route without a second deployment artifact.
• Your team has deep WSGI operational knowledge and no appetite for ASGI.

Choose FastAPI if:

• You need token streaming to a browser or mobile client.
• Expected concurrency exceeds a handful of simultaneous chats.
• You want strict input validation and auto-generated docs to iterate on prompt payloads.
• You plan to use WebSockets for a persistent chat session.

Hybrid path: Keep Flask for the legacy app, but put the LLM proxy behind a separate FastAPI service if traffic grows. Both speak HTTP, so a reverse proxy can route /api/chat accordingly.

For most new projects starting today, the flask vs fastapi llm chatbot decision leans toward FastAPI because streaming and concurrency are first-class. Flask remains a perfectly valid choice for a weekend demo or a tightly scoped internal assistant where the synchronous model is not a bottleneck.

The framework is a detail compared to your prompt design and LLM routing strategy, but it determines how painful the last 10% of production hardening will be.