Cloud Run autoscaling llm traffic demands a different mental model than scaling a typical JSON API. A single LLM request can hold an instance busy for tens of seconds while streaming thousands of tokens, so the default concurrency of 80 will exhaust memory and CPU before the autoscaler reacts. This guide lays out an ordered path to run a high-throughput LLM proxy or inference frontend on Cloud Run without falling over under real load.
1. Set concurrency and timeouts before you deploy
Cloud Run’s autoscaler adds instances based on concurrent requests (or CPU) against a per-instance cap called containerConcurrency. For LLM workloads, that cap is the single most important knob.
If you are running a pure proxy—accepting HTTP from clients and forwarding to a remote model API—your instance spends most of its time waiting on network I/O. Concurrency of 20–50 is often safe. If you are running local inference (including Cloud Run’s GPU preview), set concurrency to 1; a single generation saturates the device.
Timeouts matter because streams can run long. Set timeoutSeconds to the maximum generation time you will allow, not the default 300s if you need more.
apiVersion: serving.knative.dev/v1
kind: Service
spec:
template:
metadata:
annotations:
autoscaling.knative.dev/maxScale: "200"
autoscaling.knative.dev/minScale: "2"
spec:
containerConcurrency: 16
timeoutSeconds: 600Deploy with gcloud run deploy --concurrency=16 --timeout=600s. A common pitfall is leaving concurrency at 80 and watching OOMKills once context sizes grow. Memory scales with concurrent requests holding prompt buffers, not just with CPU.
2. Decouple upstream provider limits from instance count
Spinning up more Cloud Run instances does nothing if your upstream model provider throttles you at 100 req/min. The autoscaler will happily launch 50 instances, all of which receive 429s.
Put a token-bucket rate limiter in front of the provider call inside your service, or route through a gateway that absorbs provider variability. If you route through n4n.ai, its automatic fallback when a provider is rate-limited or degraded means a Cloud Run instance can retry against a different model without you writing provider-specific backoff. Either way, your scaling strategy must assume upstream quota is the real bottleneck.
Client-side backoff is still required. Below is a minimal async retry for an OpenAI-compatible client:
import asyncio
from openai import AsyncOpenAI
client = AsyncOpenAI(base_url="https://api.example.com/v1")
async def complete_with_backoff(prompt: str, max_retries: int = 4):
for attempt in range(max_retries):
try:
return await client.chat.completions.create(
model="proxy-model",
messages=[{"role": "user", "content": prompt}],
stream=False,
)
except Exception as e:
if "429" in str(e) and attempt < max_retries - 1:
await asyncio.sleep(min(2 ** attempt, 8))
continue
raiseTradeoff: retries increase tail latency. For chat UX, fail fast after one fallback and return a partial message instead of blocking for 30s.
3. Stream tokens and abort on client disconnect
Buffering a full LLM response in memory before replying defeats Cloud Run’s strength. Use server-sent events or chunked transfer and forward bytes as they arrive.
In FastAPI:
from fastapi import Request
from fastapi.responses import StreamingResponse
@app.post("/v1/generate")
async def generate(request: Request):
async def event_stream():
upstream = await open_upstream_stream()
async for chunk in upstream:
if await request.is_disconnected():
upstream.close()
break
yield f"data: {chunk}\n\n"
return StreamingResponse(event_stream(), media_type="text/event-stream")The is_disconnected() check is not optional. Without it, a client that navigates away leaves your instance streaming to a closed socket, wasting concurrency slots and upstream tokens. Cloud Run counts that request as concurrent until it finishes or times out.
4. Keep warm instances for spike absorption
Cold starts on Cloud Run pull your image, start the runtime, and—if you use ML frameworks—import large libraries. For a PyTorch image this can be 15–40 seconds. Under a traffic spike, the autoscaler will lag behind.
Set minScale to a value that covers your baseline p95 load:
gcloud run services update my-llm-proxy \
--min-instances=3 \
--max-instances=100Tradeoff: min instances bill continuously, even idle. For cost-sensitive dev environments, set min to 0 and accept the first-request penalty. For production cloud run autoscaling llm traffic, a small min floor is cheaper than dropped requests during a viral spike.
5. Derive concurrency from observed token throughput
Cloud Run’s autoscaler only understands concurrency or CPU. It does not know what a token is. You must translate your latency budget into a concurrency number.
Measure tokens/sec per instance during a load test. Suppose one instance sustains 2,500 tok/s and your p95 response is 600 generated tokens. Average generation time is 0.24s. If you want no request queued beyond 2s, max concurrent generations = 2.0 / 0.24 ≈ 8. Set containerConcurrency: 8.
Recompute when you change model, context size, or hardware. A 70B model on CPU will slash tokens/sec and force concurrency to 1–2.
6. Export token metrics for visibility
You cannot tune what you cannot see. Cloud Run gives request count and latency, but not token economics. Push custom metrics to Cloud Monitoring:
from google.cloud import monitoring_v3
import time
client = monitoring_v3.MetricServiceClient()
project = "projects/your-gcp-project"
def record_tokens(tokens: int):
series = monitoring_v3.TimeSeries()
series.metric.type = "custom.googleapis.com/llm/tokens_out"
series.resource.type = "cloud_run_revision"
series.resource.labels["service_name"] = "my-llm-proxy"
point = series.points.add()
point.value.int64_value = tokens
point.interval.end_time.seconds = int(time.time())
client.create_time_series(name=project, time_series=[series])Use these dashboards to spot when concurrency is too high (latency climbs) or too low (instances idle). Autoscaling still rides on concurrency, but your alerting rides on tokens.
7. Load test with real streaming and context spread
A load test that fires 100 byte prompts and reads the full body instantly will tell you nothing about cloud run autoscaling llm traffic. Use variable context lengths and actually consume the stream.
Minimal Locust task:
from locust import User, task, between
import httpx, asyncio
class LLMUser(User):
wait_time = between(1, 3)
@task
def stream(self):
async def run():
async with httpx.AsyncClient() as c:
async with c.stream("POST", "/v1/generate",
json={"prompt": "x" * 2000}) as r:
async for _ in r.aiter_text():
pass
asyncio.run(run())Run from a separate region to avoid internal bandwidth masking limits. Watch for 503s from Cloud Run when max instances cap is hit—raise it or add quota.
8. Common pitfalls and tradeoffs
- CPU allocation: By default Cloud Run bills CPU only when handling requests. If you run background cleanup (e.g., closing stale upstream connections), set
cpu-always-allocatedor the work gets paused. - Response size: Streaming avoids the 32 MB response cap, but non-streaming endpoints will 413 on large completions.
- Max instances: The platform enforces a regional quota. Request increases before launch.
- Cost: At high throughput, dedicated GPU VMs may beat Cloud Run on price, but you lose managed autoscaling. Evaluate based on steady vs bursty profile.
The ordered path is: cap concurrency to token reality, separate provider limits from instance scaling, stream and cancel, keep a warm floor, measure tokens, and load test honestly. Follow that and cloud run autoscaling llm traffic becomes a solved operations problem rather than a midnight page.