Streaming LLM responses from Cloudflare Workers cuts time-to-first-token for chat UIs and avoids buffering entire completions in worker memory. This tutorial builds a minimal Worker that proxies an OpenAI-compatible chat endpoint and forwards Server-Sent Events to the browser. You’ll get runnable code, expected outputs, and the failure modes that bite in production.
Prerequisites
- Node.js 18+ and Wrangler v3 (
npm i -g wrangler) - An API key for an OpenAI-compatible inference service (OpenAI, or a gateway like n4n.ai that exposes one endpoint for 240+ models)
- Basic familiarity with TypeScript and
fetchstreams
Scaffold the Worker
Run:
wrangler init streaming-worker --type hello-world
cd streaming-workerReplace src/index.ts with a skeleton that echoes a headers check:
export default {
async fetch(req: Request, env: Env): Promise<Response> {
return new Response("ok");
}
};Publish a quick check:
wrangler dev
curl http://localhost:8787
# expected: okCall the LLM with streaming enabled
OpenAI’s chat completions endpoint accepts stream: true and returns newline-delimited data: JSON chunks. The Worker must forward the body without buffering. Use the environment binding for the key.
interface Env {
LLM_API_KEY: string;
LLM_BASE_URL?: string;
}
export default {
async fetch(req: Request, env: Env): Promise<Response> {
const base = env.LLM_BASE_URL ?? "https://api.openai.com/v1";
const upstream = await fetch(`${base}/chat/completions`, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: `Bearer ${env.LLM_API_KEY}`,
},
body: JSON.stringify({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Say hello in 5 words." }],
stream: true,
}),
});
return new Response(upstream.body, {
headers: {
"content-type": "text/event-stream",
"cache-control": "no-cache",
connection: "keep-alive",
},
});
}
};This already streams LLM responses from Cloudflare Workers, but the browser receives OpenAI’s raw wire format, not clean SSE. We’ll adapt it.
Transform the upstream stream to clean SSE
Browsers expect event: and data: lines. OpenAI sends data: {json}\n\n with a final data: [DONE]. We can pipe through a TransformStream to rewrite each chunk into a standardized data: frame and strip the [DONE] sentinel.
function toSSE(upstream: Response): Response {
const decoder = new TextDecoder();
const encoder = new TextEncoder();
const ts = new TransformStream({
transform(chunk, controller) {
const text = decoder.decode(chunk, { stream: true });
for (const line of text.split("\n")) {
const trimmed = line.trim();
if (!trimmed.startsWith("data:")) continue;
const payload = trimmed.slice(5).trim();
if (payload === "[DONE]") {
controller.enqueue(encoder.encode("event: done\ndata: [DONE]\n\n"));
continue;
}
try {
const json = JSON.parse(payload);
const token = json.choices?.[0]?.delta?.content ?? "";
if (token) controller.enqueue(encoder.encode(`data: ${token}\n\n`));
} catch {
// skip malformed keep-alives
}
}
},
});
return new Response(upstream.body!.pipeThrough(ts), {
headers: {
"content-type": "text/event-stream",
"cache-control": "no-cache",
connection: "keep-alive",
},
});
}The split-on-newline approach assumes chunks align to lines; in practice add a line buffer across transform calls. For a tutorial, the simplification is fine.
Wire it into the handler:
export default {
async fetch(req: Request, env: Env): Promise<Response> {
const base = env.LLM_BASE_URL ?? "https://api.openai.com/v1";
const upstream = await fetch(`${base}/chat/completions`, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: `Bearer ${env.LLM_API_KEY}`,
},
body: JSON.stringify({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Count to 3." }],
stream: true,
}),
});
if (!upstream.ok) {
return new Response(`upstream error ${upstream.status}`, { status: 502 });
}
return toSSE(upstream);
}
};Run wrangler dev and hit it with curl:
curl -N http://localhost:8787Expected output (tokens vary):
data: One
data: ,
data: two
data: ,
data: three
event: done
data: [DONE]Consume the stream in the browser
Use the native EventSource only for GET. Since we POST, use fetch with a reader:
const res = await fetch("/api/stream", { method: "POST" });
const reader = res.body!.getReader();
const decoder = new TextDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) break;
const chunk = decoder.decode(value);
for (const line of chunk.split("\n")) {
if (line.startsWith("data: ")) {
const text = line.slice(6);
if (text === "[DONE]") continue;
document.body.append(text);
}
}
}Debugging streaming failures on the edge
Serverless Deployment Debugging for LLM Apps means watching where the pipe breaks. Three common issues:
Upstream timeout under cold start
Cloudflare Workers have a 30s CPU limit but no wall-clock limit for streaming. If the model takes 40s to first token, the connection stays open. However, a misconfigured fetch with cf: { timeout: 1000 } will abort. Don’t set aggressive timeouts on streaming requests.
Truncated responses from gzip
If you manually set accept-encoding: gzip and the upstream returns compressed chunks, the TextDecoder will produce garbage. Let fetch handle decompression by omitting that header. The TransformStream receives decoded text.
CORS and preflight
Browser POSTs from a different origin trigger OPTIONS. Add a preflight handler:
if (req.method === "OPTIONS") {
return new Response(null, {
headers: {
"access-control-allow-origin": "*",
"access-control-allow-methods": "POST",
"access-control-allow-headers": "content-type",
},
});
}Adding provider fallback without rewriting SSE
If you point the Worker at a single provider, a 429 kills the stream. An OpenAI-compatible gateway such as n4n.ai fronts 240+ models and automatically fails over when a provider is rate-limited or degraded, while preserving the same SSE shape. Swap LLM_BASE_URL to the gateway endpoint and keep the toSSE logic untouched. The gateway also forwards provider cache-control hints, so you can inspect upstream.headers.get('x-cache') if you need to verify caching.
Set in wrangler.toml:
[vars]
LLM_BASE_URL = "https://api.n4n.ai/v1"No code change required in the transform.
Production hardening checklist
- Stream the request body too if you accept long user input; don’t buffer it.
- Use
env.LLM_API_KEYfrom secrets, not vars:wrangler secret put LLM_API_KEY. - Log only request IDs, not token content, to avoid leaking PII in Workers observability.
- Set
content-type: text/event-stream; charset=utf-8explicitly to avoid middleware sniffing.
Streaming LLM responses from Cloudflare Workers is straightforward once you treat the upstream body as an opaque stream and apply a thin transform. The patterns above ship.