Cloud Run concurrency settings for streaming LLM requests

Cloud Run’s default concurrency of one request per instance collapses under streaming LLM workloads. Getting cloud run concurrency llm streaming right means balancing open connections, upstream token latency, and per-instance memory so you don’t either burn money on idle instances or drop streams under load.

1. How Cloud Run concurrency actually works

Cloud Run allocates each revision a concurrency value: the maximum number of simultaneous requests a single container instance will handle. The default is 1. The service scales the number of instances based on incoming request rate and concurrency setting.

For a normal JSON API, concurrency=1 is safe because requests are short. For streaming, the request stays open for the entire generation, which can be 10–60 seconds. If you leave concurrency at 1, an instance streaming a 30-second response sits at maybe 5% CPU while blocking a slot that could serve other streams.

gcloud run deploy llm-proxy \
  --image gcr.io/your-proj/llm-proxy:latest \
  --concurrency 4 \
  --cpu 1 \
  --memory 512Mi \
  --timeout 3600

The --concurrency 4 flag tells Cloud Run to pack up to four in-flight streams onto one instance. That single change often cuts cost per token by 3–4x for I/O-bound proxies.

2. Streaming LLM request anatomy

A typical streaming call looks like:

POST /v1/chat/completions
{
  "model": "gpt-4o-mini",
  "stream": true,
  "messages": [{"role": "user", "content": "Explain Raft"}]
}

Your Cloud Run service receives the request, opens a connection to the upstream model provider, and pipes Server-Sent Events back to the client. The instance is occupied for the full stream duration. The work your code does per token is trivial—parse, maybe transform, forward—but the wall-clock time is dominated by waiting on the network.

This is why cloud run concurrency llm streaming behaves more like a websocket proxy than a CRUD endpoint. You are multiplexing slow I/O, not computing.

3. Pick a starting concurrency number

Don’t guess. Measure first:

1. Log time-to-first-token (TTFT) and total-stream-duration for a representative prompt.
2. Profile CPU usage of your proxy during a single stream (usually <10% of one core).
3. Compute idle fraction: if CPU is busy 8% of the stream, the instance can theoretically handle ~12 concurrent streams.

A practical formula:

concurrency ≈ 1 / cpu_busy_fraction

But memory caps it first. A 512 MiB instance running Python with httpx and FastAPI holds roughly 30–50 MB baseline; each open stream adds a few MB of buffers. Set concurrency to 4 for 1 vCPU / 512 MiB as a safe start, then raise it after load tests.

When tuning cloud run concurrency llm streaming, always watch OOMKilled events in Cloud Run metrics. If instances restart under load, drop concurrency or raise memory.

4. Deploy a minimal streaming proxy

Below is a FastAPI service that forwards streaming requests to an OpenAI-compatible endpoint. It does not buffer the full response.

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import httpx

app = FastAPI()
UPSTREAM = "https://api.openai.com/v1/chat/completions"

@app.post("/proxy")
async def proxy(req: Request):
    body = await req.json()
    headers = {"Authorization": "Bearer " + req.headers.get("x-api-key", "")}
    async with httpx.AsyncClient(timeout=httpx.Timeout(3600)) as client:
        async with client.stream(
            "POST", UPSTREAM, json=body, headers=headers
        ) as upstream:
            async def gen():
                async for chunk in upstream.aiter_text():
                    yield chunk
            return StreamingResponse(gen(), media_type="text/event-stream")

Key points:

• client.stream and aiter_text avoid loading the body into memory.
• StreamingResponse with media_type="text/event-stream" prevents FastAPI from buffering.
• The timeout must exceed your longest expected stream, or Cloud Run’s own --timeout will kill it.

Deploy with the gcloud command from section 1, or via Terraform:

resource "google_cloud_run_v2_service" "proxy" {
  name     = "llm-proxy"
  location = "us-central1"
  template {
    containers {
      image = "gcr.io/your-proj/llm-proxy:latest"
      resources {
        cpu    = 1
        memory = "512Mi"
      }
    }
    max_instance_request_concurrency = 4
    timeout                         = "3600s"
  }
}

5. Autoscaling math

Cloud Run scales instances to satisfy:

required_instances ≈ ceil(peak_concurrent_streams / concurrency)

If you expect 200 concurrent streams at peak and run concurrency=10, you need 20 instances. Set --max-instances to cap spend and --min-instances 1 (or more) to avoid cold starts on the first stream.

gcloud run deploy llm-proxy \
  --concurrency 10 \
  --max-instances 20 \
  --min-instances 1

Scaling cloud run concurrency llm streaming requires this arithmetic upfront; otherwise you’ll either hit 503s or get a surprise bill.

6. Load test with real streams

A single curl proves the happy path. It does not prove concurrency. Write a tiny async client:

import aiohttp, asyncio

async def one(session, i):
    payload = {
        "stream": True,
        "messages": [{"role": "user", "content": f"Count to {i}"}]
    }
    async with session.post(
        "https://llm-proxy-xyz.run.app/proxy",
        json=payload,
        headers={"x-api-key": "test"}
    ) as resp:
        async for _ in resp.content:
            pass

async def main(n):
    async with aiohttp.ClientSession() as s:
        await asyncio.gather(*[one(s, i) for i in range(n)])

asyncio.run(main(50))

Run this with n=50 against a service configured for concurrency=10. Watch Cloud Run instance count climb to ~5 and confirm no 429s from your upstream.

7. Pitfalls and tradeoffs

CPU allocation mode. By default Cloud Run allocates CPU only during request handling. With streaming that’s fine—the request is active. If you switch to “CPU always allocated” for background work, you pay for idle cycles.

Client disconnects. A browser may close the tab mid-stream. Your proxy must cancel the upstream request, or you leak sockets and hit memory limits. Wrap the generator in try/finally and call upstream.aclose().

Upstream rate limits. If the provider returns 429, your instance is stuck holding client connections while you retry. A retry loop inside the stream increases memory per request. If you route through a gateway like n4n.ai, automatic fallback when a provider is rate-limited or degraded lets you keep Cloud Run concurrency high without writing custom retry and circuit-breaker code.

Response buffering in middleboxes. Some WSGI servers or reverse proxies buffer. Use ASGI (FastAPI/uvicorn) and verify with curl -N that tokens arrive incrementally.

Memory vs concurrency. Doubling concurrency on a fixed memory instance raises OOM risk linearly. Prefer vertical scaling (more memory) before pushing concurrency past 16 on a 512 MiB instance.

8. Recommended starting configuration

For a 1 vCPU, 512 MiB Cloud Run service acting as a streaming LLM proxy:

• concurrency = 4 (raise to 8 after load test)
• timeout = 600s (or 3600s if you support long agents)
• min-instances = 1 for production, 0 for dev
• max-instances = peak_streams / concurrency + headroom

Iterate: measure p99 stream duration, instance CPU, and OOM events. The right number is the one where instance CPU sits around 60–70% during peak while memory stays under 80% of limit. Cloud Run concurrency for LLM streaming is not a static setting—it’s a dial you turn with data.