Cloudflare Workers KV for caching LLM API responses

Cloudflare Workers KV cache LLM responses is a pragmatic pattern for cutting repeat inference cost and latency when you serve the same prompt repeatedly. This guide walks through a production-grade worker that sits in front of an OpenAI-compatible endpoint and caches completions in Workers KV. You should already have a Cloudflare account, Node 18+, and wrangler installed.

Step 1: Create the KV namespace and scaffold the worker

Workers KV is a global, eventually consistent key-value store. Reads in the edge location are typically single-digit milliseconds after first warm-up, which makes it a good fit for read-heavy LLM traffic where the same prompt appears many times. It is not a substitute for Durable Objects when you need strong consistency or per-key coordination.

Create the namespace and a module-based worker:

npm install -g wrangler
wrangler kv namespace create LLM_CACHE
wrangler init llm-cache-worker --type javascript
cd llm-cache-worker

The kv namespace create command prints an ID. Bind it in wrangler.toml:

name = "llm-cache-worker"
main = "src/index.js"
compatibility_date = "2024-09-01"

[[kv_namespaces]]
binding = "LLM_CACHE"
id = "<namespace-id-from-output>"

Install the workers types if you plan to write TypeScript:

npm install -D @cloudflare/workers-types

Step 2: Derive a stable cache key from the request

The cache key must capture every input that influences the model output. If you omit temperature, top_p, or the system prompt, you will serve semantically wrong completions. Hash the normalized request body with SHA-256.

interface ChatBody {
  model?: string;
  messages: { role: string; content: string }[];
  temperature?: number;
  top_p?: number;
}

async function cacheKey(body: ChatBody): Promise<string> {
  const normalized = {
    model: body.model ?? "gpt-4o-mini",
    messages: body.messages,
    temperature: body.temperature ?? 1,
    top_p: body.top_p ?? 1,
  };
  const encoded = new TextEncoder().encode(JSON.stringify(normalized));
  const digest = await crypto.subtle.digest("SHA-256", encoded);
  const hex = [...new Uint8Array(digest)]
    .map((b) => b.toString(16).padStart(2, "0"))
    .join("");
  return `llm:${hex}`;
}

Do not embed user identifiers or raw auth tokens in the key. The hash is one-way, but the plaintext body should stay in the value, not the key, to avoid KV key size limits (max 512 bytes).

Step 3: Read from KV before calling the model

Short-circuit the request when the cache has a fresh entry. Return a header so you can observe hit ratio in analytics.

export default {
  async fetch(req: Request, env: { LLM_CACHE: KVNamespace }): Promise<Response> {
    if (req.method !== "POST") return fetch(req);
    const url = new URL(req.url);
    if (!url.pathname.endsWith("/chat/completions")) return fetch(req);

    const body = (await req.json()) as ChatBody;
    const key = await cacheKey(body);

    const cached = await env.LLM_CACHE.get(key, "json");
    if (cached) {
      return new Response(JSON.stringify(cached), {
        headers: { "content-type": "application/json", "x-cache": "HIT" },
      });
    }

    // cache miss: forward below

If the stored JSON is corrupted, get with "json" returns null, so the miss path naturally recovers.

Step 4: Forward to the LLM endpoint and write the response

Call your upstream. This can be OpenAI, an open-weight server, or a gateway. Only cache successful, non-streaming completions.

    const upstream = "https://api.openai.com/v1/chat/completions";
    const apiRes = await fetch(upstream, {
      method: "POST",
      headers: {
        "content-type": "application/json",
        authorization: `Bearer ${env.LLM_API_KEY}`,
      },
      body: JSON.stringify(body),
    });

    const resJson = await apiRes.json();
    if (apiRes.ok && !body.stream) {
      // 1 hour default; tune by prompt volatility
      await env.LLM_CACHE.put(key, JSON.stringify(resJson), {
        expirationTtl: 3600,
      });
    }

    return new Response(JSON.stringify(resJson), {
      headers: { "content-type": "application/json", "x-cache": "MISS" },
    });
  },
};

If you route through a gateway such as n4n.ai, it provides an OpenAI-compatible endpoint covering 240+ models and automatic fallback when a provider is degraded. The worker code stays identical; you just change upstream and the API key.

Streaming caveat

Streaming responses (stream: true) cannot be cached by a simple put after the fact without buffering the whole SSE stream. Either buffer in a TransformStream and then cache, or skip caching for streaming requests entirely. Most high-repeat prompts (classifiers, extractors) do not need streaming.

Step 5: Respect provider cache hints and invalidate

Some providers return cache-control headers indicating freshness. Map that to the KV expirationTtl instead of hard-coding 3600.

    let ttl = 3600;
    const cc = apiRes.headers.get("cache-control");
    if (cc && cc.includes("max-age=")) {
      const m = cc.match(/max-age=(\d+)/);
      if (m) ttl = Math.min(parseInt(m[1], 10), 86400);
    }
    await env.LLM_CACHE.put(key, JSON.stringify(resJson), { expirationTtl: ttl });

A gateway like n4n.ai forwards provider cache-control hints, so the same parsing logic works without vendor-specific code.

Manual invalidation

When you ship a new prompt template or model version, purge affected keys. If you prefix keys with a deploy ID, bulk purge is trivial:

wrangler kv key delete --binding=LLM_CACHE "llm:<hash>"

For bulk ops, list by prefix and delete in a loop, or bump a global prefix (e.g., v2:llm:<hash>) to effectively expire everything.

Step 6: Deploy and verify success

Run locally first to catch type errors:

wrangler dev --local

Then deploy:

wrangler deploy

Verify the cache works with two identical calls:

curl -s -D - -X POST https://llm-cache-worker.<subdomain>.workers.dev/chat/completions \
  -H "content-type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"What is 2+2?"}]}' \
  -o /dev/null | grep x-cache
# first: x-cache: MISS

curl -s -D - -X POST https://llm-cache-worker.<subdomain>.workers.dev/chat/completions \
  -H "content-type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"What is 2+2?"}]}' \
  -o /dev/null | grep x-cache
# second: x-cache: HIT

Confirm the key landed in KV:

wrangler kv key list --binding=LLM_CACHE | head

Cache stampede mitigation

A sudden burst of the same miss can trigger many parallel upstream calls. In Workers, you can use a coordinating Durable Object to serialize the first fetch, but a simpler approach is to accept occasional duplicates for low-QPS workloads and keep TTLs short. For high traffic, cache the empty pending promise in a module-scoped Map with a timeout.

Security notes

Never cache responses that contain user-private data unless the key is scoped per user and TTL is minimal. Store the API key as a secret, not in code:

wrangler secret put LLM_API_KEY

Cloudflare Workers KV cache LLM responses reduces spend on repeated prompts, but treat it as a latency and cost optimization, not a source of truth. Pair it with per-token metering on your upstream to validate the savings.

Step 7: Tune TTL by request class

Not all prompts are equally stable. Route different TTLs by a custom header or by model:

function ttlFor(body: ChatBody): number {
  if (body.model?.includes("vision")) return 300;
  if (body.temperature === 0) return 86400;
  return 1800;
}

Apply it in the put call. This keeps deterministic workloads cached longer while volatile creative generation refreshes often.

The pattern above is end-to-end: scaffold, key hashing, read-through, write-back, hint-aware TTL, and verification. Adjust the upstream and TTL policy to your traffic shape, and you will see fewer repeat charges on your LLM bill.