Google Cloud Functions vs Cloud Run for LLM API workloads

The decision between cloud functions vs cloud run llm hosting shapes your tail latency, per-token cost, and ability to stream completions. Both sit on Google’s serverless infrastructure, yet they present distinct abstractions when you are wrapping inference endpoints with auth, retries, and caching.

Deployment Model

Cloud Functions (2nd gen) is a function-as-a-service layer that deploys a single HTTP or event handler. You write a function, point at a runtime, and Google builds the container for you. Cloud Run is a container-as-a-service layer: you supply an OCI image, and Google runs it behind a fully managed HTTP endpoint.

For an LLM proxy, the Cloud Functions handler is minimal:

# main.py — Cloud Functions (2nd gen)
import functions_framework
import os
import openai

client = openai.OpenAI(
    base_url="https://api.openai.com/v1",
    api_key=os.environ["OPENAI_API_KEY"]
)

@functions_framework.http
def chat(request):
    body = request.get_json(silent=True) or {}
    stream = body.get("stream", False)
    resp = client.chat.completions.create(
        model=body.get("model", "gpt-4o-mini"),
        messages=body["messages"],
        stream=stream
    )
    if stream:
        return resp.to_http_response()  # pseudo: stream SSE
    return {"content": resp.choices[0].message.content}

Cloud Run requires a Dockerfile and a server:

# Dockerfile — Cloud Run
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["gunicorn", "app:app", "--bind", "0.0.0.0:8080", "--workers", "1"]

# app.py — Cloud Run Flask app
from flask import Flask, request, Response
import openai, os

app = Flask(__name__)
client = openai.OpenAI(base_url="https://api.openai.com/v1", api_key=os.environ["OPENAI_API_KEY"])

@app.route("/chat", methods=["POST"])
def chat():
    body = request.json
    resp = client.chat.completions.create(model=body["model"], messages=body["messages"], stream=True)
    return Response(resp.to_streaming_response(), mimetype="text/event-stream")

Concurrency and Scaling

Cloud Functions scales per request by default. Each incoming HTTP call gets its own micro-instance unless you explicitly raise concurrency. That isolates requests but multiplies cold starts under bursty LLM traffic.

Cloud Run lets you set max-concurrency up to 1000 per instance. For LLM workloads where the heavy lifting is network I/O waiting on tokens, high concurrency means one instance can serve many simultaneous streams. You also control min instances to keep warm replicas behind a gateway.

gcloud run deploy llm-proxy \
  --image gcr.io/your-proj/llm-proxy \
  --concurrency 80 \
  --min-instances 1 \
  --max-instances 20

Cloud Functions gen2 exposes similar knobs via gcloud functions deploy --concurrency, but the mental model remains “one function, one scale graph.”

Cold Starts and Latency

Cold start is the time from zero instances to handling a request. Cloud Functions optimizes for supported runtimes (Python, Node, Go, Java, Ruby, PHP, .NET) by pre-baking language layers; a small Python function cold-starts in 300–800 ms typically. Cloud Run cold start depends on image size and entrypoint—a slim Python image is comparable, but a 2 GB image with torch installed can take 5–15 s.

For LLM streaming, time-to-first-token is dominated by the upstream model, not the serverless layer. The serverless cold start only adds to the initial connection overhead. If you need sub-second reconnect for interactive chat, run min-instances ≥ 1 on either product.

Cost Model

Both products bill on the same underlying primitive: vCPU-seconds, GiB-seconds, and number of requests. There is no separate per-invocation fee on Cloud Functions gen2; it is Cloud Run under the hood. The difference is control:

• Cloud Run lets you allocate CPU only during requests or always-on (paid), and set min instances to avoid cold starts at extra cost.
• Cloud Functions hides most of this; you pay for what runs, but you cannot keep a dedicated warm pool without deploying to Cloud Run equivalently.

Network egress to the LLM provider is billed separately by Google and is identical for both. If you route through an OpenRouter-class gateway such as n4n.ai—one OpenAI-compatible endpoint covering 240+ models with automatic fallback and per-token metering—your serverless layer is just a thin signed proxy, so the choice becomes purely about transport control.

Ergonomics and Developer Experience

Cloud Functions wins for speed of ship. One command:

gcloud functions deploy chat --runtime python312 --trigger-http --allow-unauthenticated

No Docker, no gunicorn config, no health checks. Great for a webhook that calls an LLM on form submit.

Cloud Run demands container literacy but pays back with reproducibility. You can pin system libraries, run a Rust binary, or use a custom Nginx front. For LLM apps that need background token counting, Prometheus metrics, or WebSocket upgrade, Cloud Run is the only option that doesn’t fight you.

Ecosystem and Integrations

Cloud Functions is wired into Eventarc, Pub/Sub, Storage, and Scheduler with first-class triggers. Want to summarize a file when it lands in GCS? That’s a few lines. Cloud Run integrates with the same services via push subscriptions and HTTP, but you wire the event schema yourself.

For LLM pipelines, Cloud Run’s ability to join a VPC connector and talk to a private inference node (or a gateway with private networking) is often decisive. Cloud Functions gen2 can also use VPC connector, but the configuration is less obvious in the function-centric CLI.

Limits and Quotas

Both enforce project-level quotas on instances and CPU. Cloud Run caps a single revision at 8 vCPU and 32 GiB memory; Cloud Functions gen2 inherits similar ceilings but presents them as function settings. Request timeout is 60 minutes on both, sufficient for batch embedding jobs.

Cloud Run allows HTTP/2 and WebSocket; Cloud Functions supports streaming responses but not raw WebSocket upgrades. If your client uses WebSocket to stream tokens, Cloud Run is mandatory.

Comparison Table

Which to Choose

Use Cloud Functions when

• You need a quick HTTP endpoint that calls an LLM on a schedule or from a Pub/Sub topic.
• Your team does not want to maintain Dockerfiles.
• Traffic is spiky but low-concurrency (e.g., internal Slack bot commands).
• You want tight integration with Google event sources without writing glue code.

Use Cloud Run when

• You stream tokens over HTTP/2 or WebSocket to browsers.
• You need concurrency > 10 per instance to amortize idle wait on model responses.
• You require a custom runtime, native extensions, or a sidecar for caching.
• You want explicit min-instance warm pools to guarantee p99 latency for paid users.
• You are building a multi-tenant LLM proxy that forwards routing hints and cache-control headers to an upstream gateway.

For most production LLM API workloads that outgrow a demo, cloud functions vs cloud run llm trade-offs resolve to control versus convenience. Start on Cloud Functions to validate the prompt chain; migrate the hot path to Cloud Run before you pay for idle scaled-out functions.