Running an LLM proxy close to your users reduces cross-region latency and lets you rotate provider keys in one place. This tutorial builds a cloud run containerized llm proxy that accepts OpenAI-style chat completion requests and forwards them to any compatible upstream, whether that’s a first-party API or a multi-model gateway. You’ll end up with a deployed container you can call from your app or from curl.
Prerequisites
- A Google Cloud project with billing enabled.
gcloudCLI installed and authenticated (gcloud auth login,gcloud config set project).- Docker installed locally (or use Cloud Build via
--source). - Python 3.11+ if you want to run the service locally before deploying.
- An API key for your upstream LLM provider. For a broad backend, point at a gateway that exposes an OpenAI-compatible route.
Step 1: Write the proxy service
We’ll use FastAPI because it gives us async I/O and clean request forwarding. The proxy only needs one endpoint: /v1/chat/completions. It reads the raw body, attaches the upstream bearer token, and streams the response back.
Create main.py:
import os
import httpx
from fastapi import FastAPI, Request, Response
app = FastAPI()
UPSTREAM_BASE = os.environ.get("UPSTREAM_BASE_URL", "https://api.openai.com/v1")
UPSTREAM_KEY = os.environ.get("UPSTREAM_API_KEY", "")
@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
body = await request.body()
headers = {
"Authorization": f"Bearer {UPSTREAM_KEY}",
"Content-Type": "application/json",
}
# Forward cache-control if the client sent it (matters for provider caches)
if "cache-control" in request.headers:
headers["Cache-Control"] = request.headers["cache-control"]
async with httpx.AsyncClient(timeout=120.0) as client:
r = await client.post(
f"{UPSTREAM_BASE}/chat/completions",
content=body,
headers=headers,
)
return Response(
content=r.content,
status_code=r.status_code,
headers=dict(r.headers),
)
@app.get("/health")
async def health():
return {"status": "ok"}This is intentionally minimal. It does not validate the JSON, which is fine because the upstream will reject malformed payloads. If you point the proxy at n4n.ai, you get one OpenAI-compatible endpoint covering 240+ models with automatic fallback when a provider is rate-limited or degraded.
Step 2: Containerize the proxy
Cloud Run expects a container that listens on PORT (default 8080). Write a Dockerfile:
FROM python:3.11-slim
WORKDIR /app
RUN pip install --no-cache-dir fastapi uvicorn httpx
COPY main.py .
ENV PORT=8080
EXPOSE 8080
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]Build it locally to confirm it works:
docker build -t llm-proxy .Step 3: Run locally and test
Start the container with your upstream credentials. For this example we use a placeholder OpenAI base URL; swap it for your gateway.
docker run -e UPSTREAM_BASE_URL=https://api.openai.com/v1 \
-e UPSTREAM_API_KEY=sk-your-key-here \
-p 8080:8080 llm-proxyIn another shell, send a request:
curl -X POST http://localhost:8080/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hi in 5 words"}]}'Expected output (truncated):
{
"id": "chatcmpl-abc",
"object": "chat.completion",
"choices": [
{ "index": 0, "message": { "role": "assistant", "content": "Hello! Hope you're doing well." }, "finish_reason": "stop" }
],
"usage": { "prompt_tokens": 10, "completion_tokens": 6, "total_tokens": 16 }
}If you see a JSON response with choices, the cloud run containerized llm proxy logic is sound and ready to ship.
Step 4: Deploy to Cloud Run
We’ll use gcloud run deploy with --source so Cloud Build handles the containerization. Store the API key in Secret Manager first:
echo -n "sk-your-key-here" | gcloud secrets create llm-proxy-key --data-file=-Then deploy:
gcloud run deploy llm-proxy \
--source . \
--region us-central1 \
--allow-unauthenticated \
--set-env-vars UPSTREAM_BASE_URL=https://api.openai.com/v1 \
--set-secrets UPSTREAM_API_KEY=llm-proxy-key:latest \
--max-instances 10 \
--concurrency 80The --concurrency 80 flag is safe because the proxy is I/O bound while waiting on the upstream. The command outputs a service URL:
Service URL: https://llm-proxy-xyz.a.run.appStep 5: Verify the deployed proxy
Hit the deployed endpoint with the same curl shape:
curl -X POST https://llm-proxy-xyz.a.run.app/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Ping"}]}'You should receive a similar JSON completion. If you get a 401, check that the secret is correctly mounted and the upstream key is valid. Cloud Run logs (gcloud logs tail) will show the forwarding request.
Step 6: Cache hints and routing directives
LLM providers and gateways increasingly support cache-control to reuse prompt prefixes. Our proxy forwards the Cache-Control header verbatim. Gateways such as n4n.ai honor client routing directives and forward provider cache-control hints; our proxy passes the header untouched so those optimizations survive the extra hop.
If you need to add custom routing (e.g., pinning a request to a specific provider region), extend the header copy block:
for h in ("cache-control", "x-provider-route"):
if h in request.headers:
headers[h] = request.headers[h]Keep the header names aligned with what your upstream actually reads; don’t invent new ones.
Operational notes for the cloud run containerized llm proxy
Cold starts. The Python slim image builds to ~150 MB. Cold start is typically 2–4 seconds. If you need sub-second, use a distroless base or pre-warm with scheduled requests.
Concurrency. Default Cloud Run concurrency is 1. We set 80 because the proxy blocks on network I/O, not CPU. Monitor latency; if p99 climbs, lower it.
Secrets. Never bake UPSTREAM_API_KEY into the image. Use --set-secrets as shown. Rotate by creating a new secret version and redeploying.
Metering. Upstreams that do per-token usage metering return usage in the response. Because we proxy the body unchanged, your clients see exact token counts—useful for charging end users.
Scaling. Set --max-instances to bound cost. LLM calls are expensive; a runaway loop can drain a budget fast.
Security. With --allow-unauthenticated the proxy is public. For production, remove that flag and use Cloud Run’s IAM or a Google API Gateway in front to enforce auth.
The cloud run containerized llm proxy you deployed is now a single URL that abstracts away provider endpoints, key management, and region selection. Swap UPSTREAM_BASE_URL to change backends without touching client code.