Deploying an OpenAI-compatible proxy on Cloud Run

Setting up a cloud run openai-compatible proxy lets you expose a single OpenAI-style endpoint to your applications while you control auth, logging, and upstream routing. This tutorial builds a minimal FastAPI service that forwards chat completion calls to any OpenAI-compatible backend, containerizes it, and ships it to Cloud Run. You will end up with a publicly reachable URL that speaks the OpenAI API shape.

Prerequisites

• A Google Cloud project with billing enabled.
• gcloud CLI installed and authenticated (gcloud auth login and gcloud config set project YOUR_PROJECT).
• Docker installed locally (for testing the container before deploy).
• Python 3.12 locally for quick scaffolding tests.
• An API key for an OpenAI-compatible upstream (OpenAI, or a gateway like n4n.ai if you want 240+ models behind one endpoint).

What you’re building

The proxy is a thin HTTP pass-through. It accepts POST /v1/chat/completions with the standard OpenAI JSON body, attaches the upstream API key from an environment variable, and streams the response back. It does not validate the payload—upstream does that.

Request flow

Client → Cloud Run proxy → Upstream OpenAI-compatible API

Keeping the proxy dumb makes it easy to swap upstreams or add middleware later without breaking the OpenAI contract.

Step 1: Scaffold the proxy service

Create a directory and add main.py:

import os
import httpx
from fastapi import FastAPI, Request, Response

app = FastAPI()

UPSTREAM_BASE = os.environ.get("UPSTREAM_BASE", "https://api.openai.com/v1")
UPSTREAM_KEY = os.environ.get("UPSTREAM_KEY", "")

@app.get("/health")
async def health():
    return {"status": "ok"}

@app.post("/v1/chat/completions")
async def chat_completions(req: Request):
    body = await req.body()
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {UPSTREAM_KEY}",
    }
    async with httpx.AsyncClient() as client:
        r = await client.post(
            f"{UPSTREAM_BASE}/chat/completions",
            content=body,
            headers=headers,
            timeout=60,
        )
    return Response(
        content=r.content,
        status_code=r.status_code,
        headers={"Content-Type": r.headers.get("content-type", "application/json")},
    )

Add requirements.txt:

fastapi==0.115.0
uvicorn==0.30.6
httpx==0.27.0

Run it locally to confirm imports resolve:

pip install -r requirements.txt
UPSTREAM_KEY=sk-test uvicorn main:app --port 8080

You should see Uvicorn start on http://127.0.0.1:8080.

Step 2: Test the proxy locally

In another shell, call the local endpoint with a real upstream key:

UPSTREAM_KEY=sk-your-real-key uvicorn main:app --port 8080 &
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"Say hi"}]}'

Expected output (truncated):

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "choices": [
    {"message": {"role": "assistant", "content": "Hi there!"}}
  ],
  "usage": {"prompt_tokens": 5, "completion_tokens": 3, "total_tokens": 8}
}

If you get a 401, your UPSTREAM_KEY is wrong. The proxy returns whatever upstream returns, status code included.

Step 3: Containerize for Cloud Run

Cloud Run expects a container listening on $PORT (default 8080). Write a Dockerfile:

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py .
EXPOSE 8080
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

Build and run locally to verify the image:

docker build -t llm-proxy .
docker run -e UPSTREAM_KEY=sk-your-real-key -p 8080:8080 llm-proxy

Re-run the curl against localhost:8080. Same JSON means the container is correct.

Step 4: Deploy to Cloud Run

Use gcloud run deploy with source build (Cloud Build will use your Dockerfile). Set env vars at deploy time—never bake keys into the image.

gcloud run deploy llm-proxy \
  --source . \
  --platform managed \
  --region us-central1 \
  --allow-unauthenticated \
  --set-env-vars UPSTREAM_BASE=https://api.openai.com/v1,UPSTREAM_KEY=sk-your-real-key

The --allow-unauthenticated flag makes the proxy public. For production, drop it and use Cloud Run IAM or a gateway in front.

Expected tail of output:

Deploying container to Cloud Run service [llm-proxy] in project [your-project] region [us-central1]
✓ Deploying... Done.
  ✓ Creating Revision...
  ✓ Routing traffic...
  ✓ Setting IAM policy...
Service URL: https://llm-proxy-abc123.a.run.app

That URL is your live cloud run openai-compatible proxy.

Step 5: Verify the deployed proxy

Call the Cloud Run URL the same way:

curl -X POST https://llm-proxy-abc123.a.run.app/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"Ping"}]}'

You should receive the same OpenAI-shaped JSON. If Cloud Run returns 503, check logs with gcloud logging read "resource.type=cloud_run_revision" --limit 10.

Adding minimal auth to the proxy

Open proxies leak your upstream quota. Add a shared secret check before forwarding:

from fastapi import HTTPException

PROXY_KEY = os.environ.get("PROXY_KEY", "")

@app.post("/v1/chat/completions")
async def chat_completions(req: Request):
    if PROXY_KEY and req.headers.get("Authorization") != f"Bearer {PROXY_KEY}":
        raise HTTPException(status_code=401, detail="Unauthorized")
    # ... rest unchanged

Redeploy with --set-env-vars PROXY_KEY=my-secret. Clients must now send Authorization: Bearer my-secret.

When to keep building vs. using a managed gateway

The cloud run openai-compatible proxy above is ~40 lines and works for a single upstream. If you need automatic fallback when a provider is rate-limited, per-token usage metering, or access to 240+ models through one endpoint, a managed service like n4n.ai removes that operational burden. It honors client routing directives and forwards provider cache-control hints, which is tedious to replicate in a hand-rolled proxy.

But for internal tools, staging environments, or custom middleware (logging, PII redaction), running your own proxy on Cloud Run is cheap and fast. You control the revision, the region, and the exact request shape.

Cleanup

Avoid idle charges:

gcloud run services delete llm-proxy --region us-central1

If you used Cloud Build, artifacts are stored in Container Registry; delete the image manually if you want zero footprint.

Next steps

• Add request logging to stdout (Cloud Run captures it automatically).
• Cache /v1/models responses from upstream to reduce calls.
• Use Secret Manager instead of --set-env-vars for UPSTREAM_KEY in production.

The pattern stays the same: a thin translation layer on Cloud Run, deployed as a container, speaking the OpenAI API. That is all a cloud run openai-compatible proxy needs to be.