How caching prompts cuts LLM inference costs

Prompt caching for cost reduction is the highest-leverage change most teams skip when they move from prototypes to production LLM systems. By marking stable context as cacheable, you stop paying full price for the same system prompt, schema, or document prefix on every request.

Step 1: Profile your request shape and isolate the immutable prefix

Every LLM call you make has a part that never changes between calls: the system instruction, the JSON schema you force the model to emit, the legal document you’re asking questions about, or the few-shot examples that anchor the task. That prefix is what you want the provider to cache.

Pull a representative sample of your production traffic and diff the requests. In a RAG pipeline, the retrieved context changes per query, but the system prompt and output formatter are fixed. In an agent loop, the tool definitions and memory format stay constant while the user message varies.

If the stable portion is under the provider’s minimum cacheable length—Anthropic requires at least 1024 tokens for an ephemeral cache; OpenAI auto-caches prefixes above a similar threshold—you won’t see savings. Concatenate system text, static examples, and any boilerplate until you clear that bar. Do not pad with random text; the prefix must be semantically identical on every call or the cache miss fires.

Step 2: Add explicit cache control to the cached boundary

Providers that support explicit caching need a signal for where the cacheable prefix ends. Anthropic uses a cache_control block on a content item. Through an OpenAI-compatible gateway, you send the native shape and let the gateway forward it.

Below is a raw call that caches the system text. The ephemeral type tells the provider to keep the prefix hot for the short TTL (roughly five minutes for Anthropic’s standard ephemeral cache).

curl https://api.n4n.ai/v1/messages \
  -H "Authorization: Bearer $N4N_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-3-5-sonnet",
    "max_tokens": 256,
    "system": [
      {
        "type": "text",
        "text": "You are a tax law assistant. Respond only with JSON matching schema {deduction_limit: number, citations: string[]}.",
        "cache_control": {"type": "ephemeral"}
      }
    ],
    "messages": [
      {"role": "user", "content": "What is the deduction limit for 2023 filings?"}
    ]
  }'

If you prefer the OpenAI SDK, pass the same structure via extra_body so the gateway does not strip provider-specific fields:

from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")

resp = client.chat.completions.create(
    model="anthropic/claude-3-5-sonnet",
    messages=[{"role": "user", "content": "What is the deduction limit for 2023 filings?"}],
    extra_body={
        "system": [{
            "type": "text",
            "text": "You are a tax law assistant. Respond only with JSON matching schema {deduction_limit: number, citations: string[]}.",
            "cache_control": {"type": "ephemeral"}
        }]
    },
)

The key point: the cache boundary must sit on a complete message or content block. You cannot cache half a sentence. If you place cache_control on a user message that changes every call, you gain nothing.

Step 3: Route through a gateway that preserves cache hints and meters tokens

If you talk to multiple providers, rewriting cache logic per vendor is busywork. A gateway that exposes one OpenAI-compatible endpoint and forwards cache-control hints lets you keep a single client. n4n.ai honors client routing directives and forwards provider cache-control hints, so the curl above hits Anthropic’s caching path without a separate SDK. The same request shape works for other models that ignore the field.

Per-token usage metering matters here. You need the raw input token split—cached vs. newly written—to calculate savings. A gateway that collapses everything into a single prompt_tokens count hides whether caching fired. Without that visibility, you are guessing.

Step 4: Read the usage fields to confirm cache hits

A cached prefix produces distinct usage counters. On the first request that creates the cache entry, you’ll see cache_creation_input_tokens populated and cache_read_input_tokens at zero:

{
  "usage": {
    "input_tokens": 1120,
    "output_tokens": 64,
    "cache_creation_input_tokens": 1024,
    "cache_read_input_tokens": 0
  }
}

Within the TTL, a subsequent request with the same prefix returns:

{
  "usage": {
    "input_tokens": 1120,
    "output_tokens": 58,
    "cache_creation_input_tokens": 0,
    "cache_read_input_tokens": 1024
  }
}

If cache_read_input_tokens stays zero on repeat calls, the prefix changed (maybe a timestamp sneaked into the system prompt) or you fell below the minimum length. Log these fields on every response; they are your only ground truth.

Common mistake: mutable text in the prefix

Engineers often inject "date": "2024-05-12" into the system block for traceability. That single token invalidates the cache on every new day, sometimes every request if you use a timestamp. Move such metadata to the user turn or strip it from the cached region.

Step 5: Track per-token cost and tune cache lifetime

Cached input tokens are cheaper than freshly computed ones—Anthropic and OpenAI both discount cached reads, though the exact ratio varies by model. Export the usage stream to your metrics backend and compute effective cost per request:

def cost_per_call(usage, price_per_input, price_per_cached, price_per_output):
    new_tokens = usage["input_tokens"] - usage.get("cache_read_input_tokens", 0)
    cached = usage.get("cache_read_input_tokens", 0)
    return new_tokens * price_per_input + cached * price_per_cached + usage["output_tokens"] * price_per_output

Set alerts on cache hit rate. If it drops below 80% for a workload with a stable prefix, something is mutating the prefix—a rotating request ID, current date, or unordered JSON keys. Sort and normalize any structured text before sending.

For longer-lived context (a fixed knowledge base that updates hourly), consider extended cache TTLs if your provider offers them. Ephemeral caches expire in minutes; if your traffic is sparse, you’ll pay creation cost repeatedly. Batch or warm the cache with a synthetic call if latency and hit rate demand it.

Routing fallback note

When a provider is rate-limited, a gateway with automatic fallback will route to a secondary model. Your cache hint may not apply on the fallback if that model uses a different caching scheme. Design your prefix to degrade gracefully: the request should still succeed, just without the discount on that call.

Verify success

You have working prompt caching for cost reduction when three conditions hold:

1. The cache_read_input_tokens field is non-zero on repeated calls with the same prefix.
2. Your per-token metering shows a lower effective input cost per request compared to a baseline without cache control.
3. The cached prefix length exceeds the provider minimum and contains no per-request mutable text.

Run a load test that replays 1000 captured requests against the cached configuration and 1000 against an uncached one. Diff the usage totals. If the cached run does not show a meaningful drop in billed input tokens, revisit Step 1—the prefix was not actually stable.

Caching is not a one-time toggle. As your system prompt evolves, re-measure the prefix length and cache hit rate. Treat the cache boundary like any other performance-critical code path: profile, mark, verify, repeat.