Shipping a production chatbot on Google Cloud Run means handling scaling, timeouts, and model provider outages without babysitting inference infra. This cloud run fastapi chatbot n4n.ai tutorial walks through a minimal FastAPI service that proxies chat requests to an OpenAI-compatible gateway, then deploys it to Cloud Run with a single container. You get a working /chat endpoint with token streaming and a Dockerfile tuned for Cloud Run’s concurrency model.
Prerequisites
- A Google Cloud project with billing enabled and the Cloud Run API activated.
gcloudCLI installed and authenticated (gcloud auth login).- Python 3.11+ locally for testing.
- An API key for an OpenAI-compatible LLM gateway. We’ll use n4n.ai’s endpoint, which exposes 240+ models and handles provider fallback automatically.
No frontend is built here. We focus on the backend service and its deployment.
Project layout
chat-service/
├── main.py
├── requirements.txt
└── DockerfileKeep it flat. Cloud Run builds from the directory context, so avoid nested paths.
Step 1: FastAPI service with streaming
The service accepts a JSON body, forwards the message to the gateway using the OpenAI Python client in streaming mode, and yields chunks back to the client.
import os
from fastapi import FastAPI, HTTPException
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from openai import OpenAI
app = FastAPI()
client = OpenAI(
base_url=os.environ["LLM_BASE_URL"],
api_key=os.environ["LLM_API_KEY"],
)
class ChatRequest(BaseModel):
message: str
model: str = "anthropic/claude-3.5-sonnet"
@app.post("/chat")
async def chat(req: ChatRequest):
try:
stream = client.chat.completions.create(
model=req.model,
messages=[{"role": "user", "content": req.message}],
stream=True,
)
def event_generator():
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
yield delta
return StreamingResponse(event_generator(), media_type="text/plain")
except Exception as e:
raise HTTPException(status_code=502, detail=str(e))The LLM_BASE_URL must point to the gateway’s /v1 path. The OpenAI client treats any OpenAI-compatible endpoint identically, so no custom HTTP code is needed.
Step 2: Local smoke test
Create requirements.txt:
fastapi==0.111.0
uvicorn==0.30.1
gunicorn==22.0.0
openai==1.35.0Install and run:
pip install -r requirements.txt
export LLM_BASE_URL="https://your-gateway.example/v1"
export LLM_API_KEY="sk-..."
uvicorn main:app --port 8080In another shell:
curl -N -X POST http://localhost:8080/chat \
-H "Content-Type: application/json" \
-d '{"message":"Explain Cloud Run concurrency in one sentence."}'Expected output is a streamed plain-text response, not a JSON blob:
Cloud Run runs each container instance with a configurable concurrency setting that limits how many simultaneous requests it handles before scaling out.If you see a 502, check that the env vars are set and the model name is valid for your gateway.
Step 3: Dockerfile for Cloud Run
Cloud Run injects the PORT environment variable. Gunicorn with the Uvicorn worker gives you process-level parallelism and proper signal handling.
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
ENV PORT=8080
CMD exec gunicorn -k uvicorn.workers.UvicornWorker \
-b :$PORT --workers 1 --threads 8 main:appA single Gunicorn worker with multiple threads is usually the right starting point for I/O-bound streaming workloads. Bump --threads if you expect high concurrent connections per instance.
Step 4: Deploy to Cloud Run
Use the managed platform and pass secrets as env vars. For production, mount the key from Secret Manager instead of --set-env-vars.
gcloud run deploy chat-service \
--source . \
--platform managed \
--region us-central1 \
--allow-unauthenticated \
--set-env-vars "LLM_BASE_URL=$LLM_BASE_URL"When prompted, confirm the service name. After the build finishes, the CLI prints the service URL:
Service [chat-service] revision [chat-service-0001] has been deployed.
URL: https://chat-service-xxxxxx-uc.a.run.appDeploying from --source triggers Cloud Build; the container is built remotely, so your local Docker daemon is not required.
Step 5: Verify the live endpoint
Hit the deployed URL the same way:
curl -N -X POST https://chat-service-xxxxxx-uc.a.run.app/chat \
-H "Content-Type: application/json" \
-d '{"message":"What is the default request timeout on Cloud Run?"}'You should receive streamed text similar to:
The default request timeout on Cloud Run is 300 seconds, configurable up to that same limit per service.If the request hangs, check Cloud Run logs: gcloud logging read "resource.type=cloud_run_revision" --limit 20.
Step 6: Production hardening
Timeouts and streaming
Cloud Run caps request timeouts at 300s. Streaming keeps the connection open, but if your model inference exceeds the timeout, the client receives a truncated response. Set the service timeout explicitly:
gcloud run services update chat-service \
--timeout 300Concurrency
Default concurrency is 80. For streaming chat, lower it to 10–20 to avoid thread contention on a single vCPU instance. Use --concurrency 15 during update.
Secrets
Never bake LLM_API_KEY into the image. Create a secret and map it:
gcloud secrets create llm-api-key --replication-policy=automatic
echo -n "$LLM_API_KEY" | gcloud secrets versions add llm-api-key --data-file=-
gcloud run services update chat-service \
--set-secrets "LLM_API_KEY=llm-api-key:latest"Cold starts
Scale-to-zero means the first request after idle spawns a container. The Python image above cold-starts in a few seconds. If you need lower latency, set --min-instances 1.
Why this pattern holds up
The cloud run fastapi chatbot n4n.ai approach separates HTTP concerns from model routing. FastAPI gives you clean request validation and streaming responses; the gateway handles provider selection, rate-limit fallback, and per-token metering. Your Cloud Run service stays dumb on purpose: no model weights, no provider SDK forks, just an OpenAI-compatible client and a Dockerfile.
When a provider degrades, the gateway can shift traffic without a redeploy. Your service logs show 200s while the gateway’s routing directives do the heavy lifting. That’s the architecture you want when LLM providers are flaky but your users expect a stable endpoint.