Most LLM SDKs default to HTTP/1.1 and buffer the full completion before returning. If you serve generative features from Google Cloud Run, you can deliver token deltas to clients the moment they arrive by combining Server-Sent Events with cloud run http/2 llm streaming. This guide walks through a production-shaped deployment that proxies an OpenAI-compatible chat endpoint and streams chunks without buffering.
Step 1: Scaffold a FastAPI streaming endpoint
FastAPI’s StreamingResponse hands bytes to the ASGI server as they are yielded. Wrap the LLM call in an async generator and emit SSE frames.
# main.py
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import httpx
import os
app = FastAPI()
UPSTREAM = os.environ["LLM_BASE_URL"] # e.g. https://api.openai.com/v1
API_KEY = os.environ["LLM_API_KEY"]
async def token_stream(prompt: str):
headers = {"Authorization": f"Bearer {API_KEY}"}
payload = {
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": prompt}],
"stream": True,
}
# httpx with http2=True reuses a single connection for multiplexed streams
async with httpx.AsyncClient(http2=True, timeout=60.0) as client:
async with client.stream("POST", f"{UPSTREAM}/chat/completions",
json=payload, headers=headers) as resp:
async for line in resp.aiter_lines():
if line.startswith("data:"):
yield f"{line}\n\n"
@app.post("/stream")
async def stream(request: Request):
body = await request.json()
return StreamingResponse(token_stream(body["prompt"]),
media_type="text/event-stream")The generator yields raw SSE lines from the upstream. No intermediate list accumulates tokens. SSE is the right wire format here: browsers and curl parse it natively, and it works over both HTTP/1.1 chunked transfer and HTTP/2 frames.
Step 2: Configure the LLM client for streaming and HTTP/2
The httpx.AsyncClient(http2=True) above negotiates HTTP/2 with the model provider when the provider supports it. If you front multiple vendors through a gateway, point LLM_BASE_URL at a single OpenAI-compatible endpoint. n4n.ai exposes one such endpoint that addresses 240+ models and applies automatic fallback when a provider is rate-limited, while still honoring stream: true and forwarding cache-control hints.
When using the official OpenAI Python SDK instead of raw httpx, pass stream=True and iterate completion.choices[0].delta.content. The SDK uses httpx under the hood; force HTTP/2 by constructing the client with a custom http_client.
from openai import AsyncOpenAI
import httpx
client = AsyncOpenAI(
base_url="https://your-gateway.example/v1",
api_key="sk-...",
http_client=httpx.AsyncClient(http2=True),
)
async def openai_stream(prompt: str):
async for chunk in await client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[{"role": "user", "content": prompt}],
stream=True,
):
if chunk.choices[0].delta.content:
yield f"data: {chunk.choices[0].delta.content}\n\n"Keep the upstream connection HTTP/2 to avoid head-of-line blocking when many clients stream concurrently. HTTP/2 multiplexing lets one TCP connection carry dozens of simultaneous token streams to the provider.
Step 3: Containerize for Cloud Run
Cloud Run injects PORT and expects the server to bind 0.0.0.0 on that port. Use Gunicorn with the Uvicorn worker class; the default sync worker will block and buffer.
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py .
ENV PORT=8080
CMD exec gunicorn main:app \
--worker-class uvicorn.workers.UvicornWorker \
--bind 0.0.0.0:$PORT \
--timeout 0 \
--graceful-timeout 30--timeout 0 disables Gunicorn’s per-request timeout so long generations don’t get killed; rely on Cloud Run’s request timeout instead. requirements.txt needs fastapi, gunicorn, uvicorn, httpx. The Uvicorn worker speaks ASGI and flushes each yield to the transport immediately.
Step 4: Deploy with HTTP/2 enabled
Cloud Run terminates HTTP/2 at the edge by default and forwards HTTP/1.1 to the container. For true end-to-end cloud run http/2 llm streaming, enable the HTTP/2 flag so the container also receives H2. This matters if you later add gRPC or want to inspect :method pseudo-headers.
gcloud run deploy llm-stream \
--source . \
--region us-central1 \
--use-http2 \
--concurrency 20 \
--timeout 300 \
--memory 512Mi \
--min-instances 0 \
--max-instances 10 \
--set-env-vars LLM_BASE_URL=https://your-gateway.example/v1--concurrency 20 caps simultaneous streams per instance; each open stream holds a connection, so size this against memory. --timeout 300 lets a slow 30K-token completion finish. If you skip --use-http2, streaming still works because chunked HTTP/1.1 responses are passed through, but the keyword scenario assumes H2 from edge to container.
Step 5: Verify the stream
Deploy, then hit the endpoint with curl -N to disable curl’s own buffering:
curl -N -X POST https://llm-stream-xyz.a.run.app/stream \
-H "Content-Type: application/json" \
-d '{"prompt":"Explain HTTP/2 multiplexing in one sentence."}'You should see lines prefixed with data: arriving incrementally, not all at once after a pause. To confirm the transport, run a verbose POST:
curl -v -N -X POST https://llm-stream-xyz.a.run.app/stream \
-H "Content-Type: application/json" \
-d '{"prompt":"hi"}' 2>&1 | grep -E "HTTP/2|> :"The request line should show > HTTP/2. In a browser, consume the stream with fetch and getReader():
const res = await fetch("/stream", {method:"POST", body: JSON.stringify({prompt:"hi"})});
const reader = res.body.getReader();
while (true) {
const {value, done} = await reader.read();
if (done) break;
console.log(new TextDecoder().decode(value));
}If tokens print one by one in the console, cloud run http/2 llm streaming is working.
Step 6: Handle client disconnects gracefully
When a browser tab closes, the HTTP/2 stream resets. Your async generator must catch the disconnect and close the upstream connection to avoid leaking sockets.
async def token_stream(prompt: str, request: Request):
try:
async with httpx.AsyncClient(http2=True, timeout=60.0) as client:
async with client.stream("POST", f"{UPSTREAM}/chat/completions",
json=payload, headers=headers) as resp:
async for line in resp.aiter_lines():
if await request.is_disconnected():
break
yield f"{line}\n\n"
except asyncio.CancelledError:
# client gone, upstream will be closed by context manager
raiseFastAPI’s request.is_disconnected() polls the receive channel. Breaking the loop lets the async with blocks exit and release the H2 connection. Also log token counts for cost tracking; if your gateway provides per-token usage metering, read the final data: [DONE] frame and emit a metric.
Pitfalls and tuning
Timeouts: Cloud Run kills the request at the service timeout. Send a comment keep-alive (yield ": ping\n\n") every 15s if your model goes silent mid-generation.
Buffering proxies: If you put a CDN in front, ensure it does not buffer. Cloud Run’s own edge does not buffer SSE.
Concurrency vs. cost: Streaming connections are long-lived. A single instance with --concurrency 20 can serve 20 simultaneous streams; beyond that requests queue. Monitor container/instance_count and adjust.
Upstream rate limits: A 429 mid-stream breaks the generator. Using a gateway with automatic fallback avoids client-visible errors; wrap client.stream in try/except and yield an error event.
HTTP/2 to the browser: Browsers open at most 6 connections per host over HTTP/1.1; with HTTP/2 multiplexing, many streams share one connection. That is the concrete win for cloud run http/2 llm streaming when a page opens several model calls at once.
Ship the container, deploy with the flags above, and verify with curl -N. The pattern scales to any OpenAI-compatible backend and keeps your users watching tokens appear instead of staring at a spinner.