Serving tokens from a Worker instead of a central region cuts time-to-first-byte and keeps users near the model. This guide implements cloudflare workers edge llm streaming with the OpenAI chat completions streaming protocol and a minimal Worker script you can deploy today.
Step 1: Scaffold the Worker project
Install wrangler and create a standalone project.
npm install -g wrangler
wrangler init edge-llm-stream --from-dashboard
cd edge-llm-streamThe generated wrangler.toml needs a recent compatibility date. Streaming does not require Durable Objects or KV.
name = "edge-llm-stream"
main = "src/index.js"
compatibility_date = "2024-09-23"Workers run on V8 isolates, not containers. CPU time is metered separately from wall-clock request duration, which matters for long-lived streams. A pass-through proxy uses almost no CPU because it never buffers the response body.
Step 2: Configure the upstream endpoint and secrets
Never hard-code keys in the bundle. Store them as encrypted secrets.
wrangler secret put LLM_API_KEY
wrangler secret put LLM_BASE_URLPoint LLM_BASE_URL at any OpenAI-compatible server. If you want one endpoint that addresses 240+ models with automatic fallback when a provider is degraded, set it to n4n.ai’s OpenAI-compatible gateway. The Worker forwards cache-control hints unchanged.
Read them from the env object in your handler:
export default {
async fetch(request, env) {
const { LLM_API_KEY, LLM_BASE_URL } = env;
// handler logic
}
}Rotate keys by re-running wrangler secret put with the new value; old deployments pick it up on the next cold start.
Step 3: Implement the streaming proxy handler
The chat completions endpoint accepts stream: true and returns Server-Sent Events (SSE). The simplest correct Worker forwards the request and pipes the response stream back without modification.
export default {
async fetch(request, env) {
const url = new URL(request.url);
if (url.pathname !== "/v1/chat/completions") {
return new Response("Not found", { status: 404 });
}
const upstream = `${env.LLM_BASE_URL}/v1/chat/completions`;
const body = await request.json();
body.stream = true;
body.stream_options = { include_usage: true };
const upstreamReq = new Request(upstream, {
method: "POST",
headers: {
"content-type": "application/json",
"authorization": `Bearer ${env.LLM_API_KEY}`,
...(request.headers.get("x-cache-control")
? { "cache-control": request.headers.get("x-cache-control") }
: {})
},
body: JSON.stringify(body)
});
const upstreamRes = await fetch(upstreamReq);
return new Response(upstreamRes.body, {
status: upstreamRes.status,
headers: {
"content-type": "text/event-stream",
"cache-control": "no-cache",
"connection": "keep-alive",
...(upstreamRes.headers.get("x-ratelimit-remaining")
? { "x-ratelimit-remaining": upstreamRes.headers.get("x-ratelimit-remaining") }
: {})
}
});
}
}This gives you cloudflare workers edge llm streaming with zero token buffering. The upstreamRes.body is a ReadableStream that the runtime streams directly to the client.
Why pipe the body directly
Cloudflare exposes the upstream ReadableStream as upstreamRes.body. Returning it directly avoids copying chunks through your isolate’s memory. Latency is dominated by the model’s generation speed, not the edge hop. Avoid await upstreamRes.text() or json() — that defeats streaming.
Step 4: Parse and transform the stream (optional)
Sometimes you need to rewrite tool calls, strip internal thinking tokens, or merge delta chunks. Consume the SSE stream and emit a cleaned version.
function parseSSE(stream) {
const reader = stream.getReader();
const decoder = new TextDecoder();
let buffer = "";
return new ReadableStream({
async pull(controller) {
const { done, value } = await reader.read();
if (done) {
controller.close();
return;
}
buffer += decoder.decode(value, { stream: true });
const blocks = buffer.split("\n\n");
buffer = blocks.pop();
for (const block of blocks) {
const trimmed = block.replace(/^data: /, "").trim();
if (!trimmed || trimmed === "[DONE]") continue;
try {
const json = JSON.parse(trimmed);
if (json.choices?.[0]?.delta && Object.keys(json.choices[0].delta).length === 0) continue;
controller.enqueue(new TextEncoder().encode(`data: ${JSON.stringify(json)}\n\n`));
} catch {
controller.enqueue(new TextEncoder().encode(block + "\n\n"));
}
}
}
});
}Wire it in by replacing the return statement:
const cleaned = parseSSE(upstreamRes.body);
return new Response(cleaned, {
status: 200,
headers: { "content-type": "text/event-stream", "cache-control": "no-cache" }
});This pattern keeps the Worker in the loop per chunk, so CPU time rises slightly but stays low for typical token sizes.
Step 5: Add timeouts and error handling
Edge platforms enforce a CPU limit (typically 10–50 ms on free tiers, more on paid) but allow long wall-clock streams. The initial fetch to the model should fail fast if the upstream is unreachable.
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 10000);
let upstreamRes;
try {
upstreamRes = await fetch(upstreamReq, { signal: controller.signal });
} catch (e) {
return new Response(JSON.stringify({ error: "upstream timeout" }), { status: 504 });
} finally {
clearTimeout(timeout);
}
if (!upstreamRes.ok) {
return new Response(upstreamRes.body, {
status: upstreamRes.status,
headers: { "content-type": upstreamRes.headers.get("content-type") || "application/json" }
});
}Surface the upstream error body so clients see the real rate-limit or auth message instead of a generic 500.
Step 6: Add CORS for browser clients
If a frontend calls the Worker directly, set CORS headers on the streaming response.
const cors = {
"access-control-allow-origin": "*",
"access-control-allow-methods": "POST, OPTIONS",
"access-control-allow-headers": "content-type, authorization"
};
if (request.method === "OPTIONS") {
return new Response(null, { status: 204, headers: cors });
}Merge cors into the response headers from Step 3 or Step 4. Without this, browsers block the stream.
Step 7: Deploy and verify success
Publish the Worker:
wrangler deployThe CLI prints a *.workers.dev URL. Verify streaming with curl using -N (no buffering):
curl -N https://edge-llm-stream.<subdomain>.workers.dev/v1/chat/completions \
-H "content-type: application/json" \
-d '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"Say hi in 3 words."}]}'You should see incremental data: {...} lines ending with data: [DONE]. If you receive a single buffered blob, confirm stream:true is in the forwarded body and that the client uses -N.
For a browser check, open the console and run:
const res = await fetch("/v1/chat/completions", {
method: "POST",
body: JSON.stringify({ model: "gpt-3.5-turbo", messages: [{ role: "user", content: "Hi" }] })
});
const reader = res.body.getReader();
while (true) {
const { done, value } = await reader.read();
if (done) break;
console.log(new TextDecoder().decode(value));
}Tokens appear as they generate. That confirms cloudflare workers edge llm streaming is working end to end.
Operational notes
- Set
stream_options.include_usageto get a final usage chunk; some gateways meter per token and emit it there. - If you route between providers via client headers, forward
x-routing-directiveto the upstream so the gateway honors it. - Cloudflare never caches
POSTrequests, so streaming endpoints stay dynamic by default. - Monitor Worker CPU time, not wall-clock. A pure proxy stays well under limits even for multi-minute generations.
- For production, put the Worker behind a custom domain and add WAF rules to throttle abusive clients.
That is a complete path from zero to a deployed edge streaming proxy you can point at any OpenAI-compatible backend.