The cloudflare workers streams api llm tokens approach turns a Worker into a thin, low-latency proxy that forwards model output to clients the instant each token is generated. Instead of buffering a full completion, you pipe the provider’s Server-Sent Events (SSE) through Web Streams primitives and emit cleaned token frames. This guide walks through a production-shaped implementation on Cloudflare Workers, from wrangler init to a verified curl stream.
Step 1: Scaffold the Worker project
Start with a TypeScript Worker. Wrangler handles bundling and the Streams API is available globally at runtime—no polyfills needed.
npm create cloudflare@latest llm-stream-worker
cd llm-stream-worker
wrangler init --from-dashPick the “Hello World” Worker template (ES modules format). Your src/index.ts will export a fetch handler. Add a secret for the upstream API key:
wrangler secret put UPSTREAM_API_KEYStore the base URL and model in wrangler.toml or env vars. For local dev, wrangler dev serves on localhost:8787.
Step 2: Issue a streaming request to the LLM
Cloudflare Workers use the standard fetch API. Set stream: true in the chat completion body. The upstream must be OpenAI-compatible; if you route through n4n.ai, the same endpoint works across 240+ models with automatic fallback when a provider is degraded.
interface Env {
UPSTREAM_API_KEY: string;
UPSTREAM_BASE: string; // e.g. https://api.openai.com/v1 or https://api.n4n.ai/v1
MODEL: string;
}
export default {
async fetch(req: Request, env: Env): Promise<Response> {
const upstreamResp = await fetch(`${env.UPSTREAM_BASE}/chat/completions`, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: `Bearer ${env.UPSTREAM_API_KEY}`,
},
body: JSON.stringify({
model: env.MODEL,
stream: true,
messages: [{ role: "user", content: "Explain backpressure in 3 sentences." }],
}),
});
if (!upstreamResp.ok || !upstreamResp.body) {
return new Response("Upstream error", { status: 502 });
}
// Stream wiring continues below
},
};The cloudflare workers streams api llm tokens pattern depends on upstreamResp.body being a ReadableStream<Uint8Array>. If the provider returns non-streaming JSON, this code will not work—verify the stream: true flag is honored.
Step 3: Parse SSE with a TransformStream
OpenAI-style streams are newline-delimited data: {json} frames. A TransformStream lets you decode bytes, split lines, and extract delta.content. The transform respects backpressure: if the client is slow, controller.enqueue blocks until the downstream catches up.
function parseTokenStream(): TransformStream<Uint8Array, Uint8Array> {
const decoder = new TextDecoder();
const encoder = new TextEncoder();
let buffer = "";
return new TransformStream({
transform(chunk, controller) {
buffer += decoder.decode(chunk, { stream: true });
const lines = buffer.split("\n");
buffer = lines.pop() ?? ""; // keep partial line
for (const line of lines) {
const trimmed = line.trim();
if (!trimmed.startsWith("data: ")) continue;
const payload = trimmed.slice(6).trim();
if (payload === "[DONE]") return;
try {
const json = JSON.parse(payload);
const token: string | undefined = json.choices?.[0]?.delta?.content;
if (token) controller.enqueue(encoder.encode(token));
} catch {
// ignore malformed keep-alive lines
}
}
},
flush(controller) {
if (buffer.trim().length) {
// attempt final parse on disconnect
}
},
});
}Pipe the upstream body through this transform and re-encode for the browser:
const tokenStream = upstreamResp.body
.pipeThrough(parseTokenStream());
return new Response(tokenStream, {
headers: {
"content-type": "text/plain; charset=utf-8",
"cache-control": "no-store",
"x-accel-buffering": "no", // disable proxy buffering
},
});Step 4: Return a client-safe stream
Browsers and fetch clients treat the response body as a stream automatically. Set content-type to text/plain or application/octet-stream. Avoid text/event-stream unless you are forwarding raw SSE—our transform already extracted tokens, so a flat byte stream is simpler to consume with response.body.getReader().
If you need JSON frames per token, change the enqueue to encoder.encode(JSON.stringify({ token }) + "\n"). That keeps the cloudflare workers streams api llm tokens output parseable line-by-line.
Step 5: Handle cancellation and errors
Workers abort the fetch handler when the client disconnects. Pass an AbortSignal to the upstream request so the provider stops generating:
const ac = new AbortController();
req.signal.addEventListener("abort", () => ac.abort());
const upstreamResp = await fetch(url, {
method: "POST",
headers,
body: JSON.stringify({ model, stream: true, messages }),
signal: ac.signal,
});Inside the TransformStream, wrap controller.enqueue in try/catch. If the downstream reader cancels, the stream will throw and you should swallow it. Never await long-running work in transform—keep it synchronous to avoid head-of-line blocking.
For auth failures, check upstreamResp.status before piping. If you get a 429, the Worker can retry with a different model or return a 503. The cloudflare workers streams api llm tokens design keeps the Worker stateless, so retries are just new fetches.
Step 6: Deploy and verify
Publish the Worker:
wrangler deployThen hit the endpoint with a raw HTTP client that disables buffering:
curl -N https://llm-stream-worker.your-subdomain.workers.dev/You should see tokens printed one by one with no delay between the full response. If you get a single buffered blob, check that x-accel-buffering: no is set and that no CDN in front of the Worker is aggregating chunks.
To verify programmatically, use a small Node script:
const resp = await fetch("https://llm-stream-worker.your-subdomain.workers.dev/");
const reader = resp.body!.getReader();
const decoder = new TextDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) break;
process.stdout.write(decoder.decode(value));
}If tokens arrive incrementally and the process exits after [DONE] (or stream close), the pipeline is correct.
Why not use a higher-level SDK?
The AI SDKs are convenient but hide the stream lifecycle. When you need to add rate-limit headers, inject per-token logging, or fork the stream to a second sink (e.g., a metrics pipe), the raw cloudflare workers streams api llm tokens primitives are clearer. TransformStream is also CPU-cheap: no intermediate string arrays beyond the line buffer, and V8 optimizes the TextDecoder path.
One caveat: Workers have a 30-second CPU limit on the free tier and 5 minutes on paid. Streaming responses do not count against CPU time while waiting on I/O, but your transform runs on the event loop per chunk. Keep parsing minimal—no regex heavier than split and startsWith.
Production notes
- Set
cache-control: no-storeon the response; LLM output is user-specific. - If you forward provider cache-control hints, honor them on the Worker response to leverage edge caching for identical prompts.
- Use
env.MODELindirection so you can shift traffic between models without redeploying. - For multi-tenant auth, validate a JWT in the Worker before calling the upstream; do not expose your
UPSTREAM_API_KEYto the browser.
The pattern scales horizontally because each Worker instance holds only open sockets and a few kilobytes of stream state. That is the entire footprint for token-by-token LLM delivery.