How to cut agent API costs by 60% with prompt caching

Engineers building LLM agents burn tokens replaying the same system prompt, tool schemas, and few-shot examples on every turn. Prompt caching cost reduction is the fastest win available: store the static prefix once and pay a fraction for subsequent reads. Here is how to implement it end to end in a production agent.

Step 1: Audit your agent’s token flow

Before changing code, measure where tokens go. Most agents send an identical block of text—system instructions, tool definitions, retrieval context—on every request, then append a small dynamic user message. That repeated block is your caching target.

Count tokens with the provider’s tokenizer. For OpenAI models, tiktoken works:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")
system_prompt = open("system_prompt.txt").read()
tool_schema = open("tools.json").read()

static_tokens = len(enc.encode(system_prompt + tool_schema))
print(f"Static prefix: {static_tokens} tokens")

If your static prefix exceeds 1,000 tokens, you are leaving money on the table. Prompt caching cost reduction starts by isolating that prefix so the provider can cache it.

Step 2: Restructure prompts to maximize cacheable prefix

Providers cache from the start of the prompt up to a marked breakpoint. Anything after the breakpoint is dynamic and bypasses the cache. Reorder your messages so the static content is first and contiguous.

A correct layout for a tool-using agent:

[
  {"role": "system", "content": "You are a DevOps assistant. Use tools to query logs and deploy."},
  {"role": "system", "content": "[TOOL DEFINITIONS]\n{...large JSON schema...}"},
  {"role": "user", "content": "What is the error rate on prod-1?"}
]

Do not interleave static and dynamic content. If you inject a timestamp into the system prompt, you invalidate the cache every call. Generate timestamps inside the user turn instead.

Step 3: Enable cache control on supported providers

Anthropic: explicit breakpoints

Anthropic requires a cache_control marker on the last block of the static prefix. The SDK accepts it as an extra field:

import anthropic

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    system=[
        {"type": "text", "text": SYSTEM_PROMPT},
        {"type": "text", "text": TOOL_SCHEMA, "cache_control": {"type": "ephemeral"}}
    ],
    messages=[{"role": "user", "content": user_query}]
)

The cache_control tells Anthropic to cache everything up to that text block. Subsequent calls with the same prefix return cache_read_input_tokens in the usage.

OpenAI: implicit prefix caching

OpenAI applies caching automatically for prompts longer than 1,024 tokens on supported models (e.g., gpt-4o, gpt-4o-mini). No API change is needed beyond keeping the prefix identical. You still benefit from the restructuring in Step 2.

Step 4: Route through a gateway that honors cache directives

If you serve multiple models or need fallback, use a gateway that forwards cache hints without stripping them. n4n.ai exposes an OpenAI-compatible endpoint across 240+ models and forwards provider cache-control hints, so the same client code works whether the backend is Anthropic or OpenAI. It also meters per-token usage, which makes verification trivial.

Point your OpenAI client at the gateway:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",
    api_key="YOUR_KEY"
)
# Same chat.completions call as OpenAI; cache hints pass through

Automatic fallback kicks in when a provider is rate-limited, but cached prefixes only apply to the original provider—expect a cold cache on fallback.

Step 5: Verify cache hits and measure savings

Never assume caching works. Check the usage object on every response.

For Anthropic:

print(response.usage)
# {'input_tokens': 120, 'output_tokens': 80, 'cache_read_input_tokens': 5400, 'cache_creation_input_tokens': 0}

For OpenAI via the gateway, inspect response_headers for x-cache-read-tokens if exposed, or rely on the prompt_tokens_details field:

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=messages
)
print(resp.usage.prompt_tokens_details)
# {'cached_tokens': 5400, 'audio_tokens': 0}

A successful rollout shows cache_read_input_tokens (or cached_tokens) growing on the second turn of a session. To verify prompt caching cost reduction, compare total input tokens over a day before and after. If your static prefix is 5,000 tokens and you process 1,000 agent turns, uncached cost is ~5M input tokens; cached, you pay creation once plus ~5K reads per turn at a 90% discount. That math yields the 60%+ cut.

Step 6: Handle multi-turn conversations and invalidation

Caching is prefix-based. In a multi-turn agent loop, append new assistant and user messages after the static block. Never mutate the system prompt mid-session.

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "system", "content": TOOL_SCHEMA}
]
while True:
    user_msg = get_user_input()
    messages.append({"role": "user", "content": user_msg})
    resp = client.chat.completions.create(model="gpt-4o", messages=messages)
    messages.append({"role": "assistant", "content": resp.choices[0].message.content})

Cache TTLs are short—Anthropic ephemeral caches last 5 minutes, OpenAI’s cached prefix persists based on recent usage. If your agent idles beyond the TTL, the next call pays creation cost again. For long-running workers, send a keepalive request with the static prefix every few minutes.

Step 7: Monitor cache miss rate in production

Log the ratio of cache_read_input_tokens to total input tokens per request. A miss rate above 10% on steady sessions signals prefix drift or TTL expiry.

import logging

def log_cache_ratio(usage):
    read = getattr(usage, "cache_read_input_tokens", 0)
    total = usage.input_tokens
    if total > 0:
        logging.info("cache_hit_ratio=%.2f", read / total)

Alert when the rolling average drops below 0.8. The usual culprits: a dynamic variable leaked into the system prompt, or a gateway routing to a model that doesn’t support caching for that prefix length.

What to watch out for

Cache creation is not free. Providers charge a small premium (often 1.25×) for writing the prefix. If your agent turns are sparse—say one call per hour—the cache expires before reuse and you lose money. Cache only when you expect at least two hits within the TTL.

Also, not all models cache equally. Smaller models may have tighter minimum prefix lengths. Test with your actual system prompt, not a toy string.

Prompt caching cost reduction is a structural change, not a toggle. Get the prefix right, verify the hits, and your agent bill drops without touching model quality.