How to keep LLM connections alive on Cloudflare Workers

When you deploy an LLM proxy on Cloudflare Workers, the default request model fights you: the platform expects short-lived subrequests, but you need to keep LLM connection alive Cloudflare Workers while tokens stream for seconds or minutes. A naive fetch that buffers the response will hit CPU limits, drop the stream, or return truncated JSON. This guide shows the exact steps to build a Worker that holds an open, resilient stream to an OpenAI-compatible endpoint and pipes tokens to the browser without dying.

Step 1: Map the runtime constraints before writing code

Cloudflare Workers run on V8 isolates with a hard 50 ms CPU time limit on the free tier and up to 30 s of wall-clock on paid plans (CPU time still capped per tick). The isolate freezes when awaiting I/O, but the connection to the LLM provider must remain open across many event loop turns. You cannot use Node’s http module or long-lived TCP sockets; you only get fetch and Web Streams. Understand three limits that bite LLM workloads:

• Subrequest count limit (default 50 per request) — a retrying proxy can blow this silently.
• Maximum response body size — streaming avoids buffering, but arrayBuffer() will kill you.
• No manual Connection: keep-alive header control; the platform manages socket pooling internally.

If you ignore these, you will see Error 1102 (Worker exceeded CPU) or silently truncated SSE streams that look like model hallucinations.

Step 2: Use streaming fetch with duplex: 'half' and zero buffering

To keep LLM connection alive Cloudflare Workers, read the response body as a stream and forward it immediately. Never call await response.json() or await response.text(). Use response.body (a ReadableStream). For streaming request bodies (sending a large prompt), set duplex: 'half' in the fetch init — this is required on Workers for request streaming.

export default {
  async fetch(req: Request): Promise<Response> {
    const upstream = await fetch("https://api.openai.com/v1/chat/completions", {
      method: "POST",
      headers: {
        "content-type": "application/json",
        "authorization": `Bearer ${OPENAI_KEY}`,
      },
      body: JSON.stringify({
        model: "gpt-4o-mini",
        messages: [{ role: "user", content: "Stream a poem." }],
        stream: true,
      }),
      // Required for streaming request bodies in Workers
      duplex: "half",
    });

    // Pipe upstream SSE straight to client
    return new Response(upstream.body, {
      headers: {
        "content-type": "text/event-stream",
        "cache-control": "no-cache",
        "connection": "keep-alive",
      },
    });
  },
};

This minimal Worker solves the core problem: the isolate never buffers the LLM response, so CPU time stays near zero while bytes flow. The connection: keep-alive header is advisory to the browser; the Worker itself stays alive because the response stream is attached to the client request.

Step 3: Add an AbortController and a timeout guard

A hung upstream connection will keep your Worker alive indefinitely on paid plans, but clients may disconnect. Wrap the fetch in an AbortController with a timeout, and listen to the client req.signal to cancel upstream if the browser closes. This is critical to keep LLM connection alive Cloudflare Workers without leaking resources or racking up billable milliseconds.

async fetch(req: Request): Promise<Response> {
  const ctrl = new AbortController();
  const timeout = setTimeout(() => ctrl.abort(), 25_000);
  req.signal.addEventListener("abort", () => ctrl.abort());

  try {
    const upstream = await fetch("https://api.openai.com/v1/chat/completions", {
      method: "POST",
      headers: { "content-type": "application/json", authorization: `Bearer ${KEY}` },
      body: JSON.stringify({ model: "gpt-4o-mini", messages: [{role:"user",content:"Hi"}], stream: true }),
      duplex: "half",
      signal: ctrl.signal,
    });
    clearTimeout(timeout);
    return new Response(upstream.body, {
      headers: { "content-type": "text/event-stream", "cache-control": "no-cache" },
    });
  } catch (e) {
    clearTimeout(timeout);
    return new Response("upstream aborted", { status: 504 });
  }
}

The req.signal abort propagates to the upstream fetch, closing the socket promptly. Without this, a user navigating away leaves an orphaned stream that counts against your concurrent connection limits.

Step 4: Handle provider degradation with fallback routing

Single-provider streams fail in production — rate limits, 529s, or regional blips. If you front your calls with a gateway like n4n.ai, which provides an OpenAI-compatible endpoint across 240+ models with automatic fallback when a provider is degraded, you get retry logic at the edge without custom code. Otherwise, implement a simple ordered list of base URLs and try each until one returns a 200 with a body.

const PROVIDERS = [
  "https://api.openai.com/v1",
  "https://openrouter.ai/api/v1",
];

async function streamWithFallback(payload: any, signal: AbortSignal) {
  for (const base of PROVIDERS) {
    try {
      const r = await fetch(`${base}/chat/completions`, {
        method: "POST",
        headers: { "content-type": "application/json", authorization: `Bearer ${KEY}` },
        body: JSON.stringify({ ...payload, stream: true }),
        duplex: "half",
        signal,
      });
      if (r.ok && r.body) return r;
    } catch (_) { /* try next */ }
  }
  throw new Error("all providers failed");
}

This pattern keeps the user-visible stream alive even when the primary LLM vendor flaps. Note that you should not retry after the first token has been sent to the client — only the initial connection attempt is safe to fail over.

Step 5: Parse and transform the SSE stream safely

Browsers expect data: {json}\n\n frames. If you need to inject metadata (token counts, model id), use a TransformStream rather than accumulating. Below we count token chunks without buffering the whole response.

function makeTransformer() {
  let tokens = 0;
  return new TransformStream({
    transform(chunk, controller) {
      // chunk is Uint8Array; in practice parse line by line
      const text = new TextDecoder().decode(chunk);
      if (text.includes("\"delta\"")) tokens++;
      controller.enqueue(chunk);
    },
    flush() {
      console.log(`streamed ${tokens} token chunks`);
    },
  });
}

// inside fetch:
const upstream = await streamWithFallback(payload, ctrl.signal);
const body = upstream.body!.pipeThrough(makeTransformer());
return new Response(body, { headers: { "content-type": "text/event-stream" } });

Proper SSE line splitting requires a small string buffer for partial lines across chunk boundaries; the snippet above is minimal but shows the streaming hook point. Never run JSON.parse on the entire accumulated string — that is how you blow the CPU limit.

Step 6: Verify the connection stays open locally

Use wrangler dev and a raw curl with --no-buffer to confirm tokens arrive incrementally and the process does not exit early.

wrangler dev --local
# in another shell
curl -N -X POST http://localhost:8787/ \
  -H "content-type: application/json" \
  -d '{"prompt":"count to 5 slowly"}'

Success criteria:

• You see data: ... lines printed one by one with delays, not all at the end.
• The Worker logs show no Error 1102.
• Pressing Ctrl-C on curl triggers the req.signal abort and the Worker logs upstream aborted (or similar) without hanging.

For production, deploy with wrangler deploy and watch Workers analytics for subrequest errors. A healthy stream shows a single subrequest to the LLM provider with duration matching the token generation time.

Step 7: Debug dropped connections in the wild

When a connection dies, the symptom is a client-side fetch that ends with net::ERR_INCOMPLETE_CHUNKED_ENCODING. On Workers, check:

1. CPU limit: if you parse JSON synchronously per chunk, you may exceed 50 ms. Offload parsing to a TransformStream and avoid JSON.parse on large arrays.
2. Response size: streaming bypasses the 100 MB limit only if you never buffer; using await response.arrayBuffer() will kill you.
3. Client timeouts: mobile browsers may close after 60 s. Send periodic : ping comments in the SSE stream to keep LLM connection alive Cloudflare Workers and the client happy.

function pingStream() {
  return new TransformStream({
    start(controller) {
      const timer = setInterval(() => {
        controller.enqueue(new TextEncoder().encode(": ping\n\n"));
      }, 15_000);
      // clear interval on cancel omitted for brevity
    },
  });
}

Attach pipeThrough(pingStream()) before returning the response. The colon prefix marks a comment in SSE, so it is ignored by parsers but resets idle timers.

Step 8: Production hardening checklist

• Set compatibility_date to at least 2023-12-01 in wrangler.toml for duplex: 'half' support.
• Store API keys in Workers environment variables, never inline.
• If you need per-token metering, a gateway that honors provider cache-control hints and returns usage in the final SSE comment simplifies billing. (n4n.ai does this, but you can also parse the usage field from the last chunk yourself.)
• Add a Retry-After handler for 429s: backoff and reconnect the stream, not just the initial request.
• Test with artillery to simulate 100 concurrent streams; ensure CPU milliseconds per request stay under plan limits.

Following these steps, you will keep LLM connection alive Cloudflare Workers reliably, stream tokens to users with low latency, and avoid the classic serverless truncation bugs that waste hours during incident reviews.