Prompt caching pricing: how discounts actually work

Prompt caching pricing discount structures vary significantly across providers, and understanding the mechanics is essential for any team running LLMs at scale. Most engineers assume cached tokens are simply cheaper, but the reality involves distinct token accounting rules, cache lifetime constraints, and routing behaviors that directly affect your bill. This analysis breaks down how the major providers implement prompt caching discounts, where the hidden costs live, and how to architect for maximum savings without sacrificing latency or correctness.

How prompt caching discounts actually work

At its core, prompt caching separates your input into two token streams: the cached prefix and the dynamic suffix. Providers charge full price for the suffix (the part that changes per request) and a discounted rate for the prefix (the part that matches a previously seen context). The discount magnitude and the conditions for eligibility differ by provider.

OpenAI applies a 50% discount on cached input tokens for models that support it (GPT-4o, GPT-4o-mini, o1-preview, o1-mini). The cache key is the exact prefix of the conversation — system prompt, few-shot examples, and any prior turns that match character-for-character. A cache hit requires an exact prefix match of at least 1,024 tokens. There is no partial credit; you either hit the full cached prefix or you don’t.

Anthropic takes a different approach with Claude 3.5 Sonnet and Haiku. They offer a 90% discount on cached tokens, but the minimum cacheable unit is 1,024 tokens for Sonnet and 2,048 tokens for Haiku. The cache key includes the entire prompt up to the cache control breakpoint you explicitly set. This gives you more control but requires deliberate prompt engineering.

Google’s Gemini 1.5 Pro and Flash offer context caching as a separate resource you create explicitly. You pay a storage fee per million tokens per hour (roughly $0.025/MTok/hr for 1.5 Pro) plus a reduced generation rate. The cache persists until you delete it or it expires (default TTL 1 hour, configurable up to 24 hours). This model shifts the economics: you pay for cache residency, not just per-request discounts.

Token accounting mechanics

The devil lives in how providers count tokens for cached vs. uncached portions. Consider this request structure:

System prompt (500 tokens)
+ Few-shot examples (800 tokens)
+ User message (200 tokens)
= 1,500 total input tokens

With OpenAI, if the first 1,300 tokens match a cached prefix, you pay full price for 200 tokens and 50% price for 1,300 tokens. But if your system prompt changes by a single character — say, a timestamp or version string — the entire 1,500 tokens bill at full rate. The cache key is brittle by design.

Anthropic’s explicit cache control breakpoints let you isolate the stable portion:

messages = [
    {"role": "user", "content": [
        {"type": "text", "text": system_prompt, "cache_control": {"type": "ephemeral"}},
        {"type": "text", "text": few_shot_examples, "cache_control": {"type": "ephemeral"}},
        {"type": "text", "text": user_message}
    ]}
]

Only tokens before the last cache_control breakpoint are eligible. This prevents accidental cache misses from minor changes downstream, but it requires you to structure prompts deliberately.

Google’s explicit cache creation makes the accounting transparent:

from google import genai

client = genai.Client()
cache = client.caches.create(
    model="gemini-1.5-pro-001",
    contents=[system_prompt, few_shot_examples],
    ttl="3600s"  # 1 hour
)

# Later requests reference the cache
response = client.models.generate_content(
    model="gemini-1.5-pro-001",
    contents=[user_message],
    cached_content=cache.name
)

You pay for the cache object’s existence regardless of request volume. At low request rates, the storage fee exceeds the per-token savings. At high volumes, the economics flip dramatically.

Cache lifetime and eviction policies

Prompt caching pricing discount value depends entirely on cache hit rate, which in turn depends on lifetime and eviction behavior. None of the major providers guarantee cache persistence.

OpenAI caches are ephemeral with undocumented TTLs (observed 5-10 minutes of inactivity). They’re also scoped to the organization and model — switching models or organizations invalidates the cache. High-traffic applications naturally keep caches warm; bursty workloads see cold starts on every burst.

Anthropic’s ephemeral caches last approximately 5 minutes of inactivity. They also support persistent caches (beta) with 1-hour default TTL, extendable to 24 hours. Persistent caches incur a storage fee similar to Google’s model but at a lower per-token discount (50% vs 90% for ephemeral).

Google gives you explicit TTL control up to 24 hours. The cache is a first-class resource you manage. This is operationally simpler for predictable workloads (daily batch jobs, scheduled evaluations) but adds infrastructure overhead for ad-hoc chat applications.

Routing and fallback interactions

If you route requests across multiple providers or models — a common pattern for cost optimization and resilience — prompt caching discounts fragment. A cache hit on OpenAI GPT-4o doesn’t transfer to Anthropic Claude or to OpenAI o1-mini. Each provider-model combination maintains independent caches.

This creates a tension: routing for lowest per-token cost may route away from your warm cache, increasing effective cost despite lower headline pricing. A request that hits a 50% cached prefix on GPT-4o often costs less than the same request at full price on a cheaper model.

# Simplified routing logic that accounts for cache state
async def route_request(messages, cache_hints):
    # Check if we have a warm cache for the primary model
    primary_cache_key = compute_prefix_hash(messages, CACHE_THRESHOLD)
    cache_warm = await cache_store.exists(primary_cache_key)
    
    if cache_warm:
        # Stay on primary model to capture discount
        return await call_primary(messages)
    
    # No cache benefit — route to cheapest available model
    return await call_cheapest_available(messages)

n4n.ai forwards provider cache-control hints and honors client routing directives, which lets you implement this logic at the gateway layer without baking provider-specific cache awareness into every service.

Quantifying the break-even point

The prompt caching pricing discount only pays off when your cache hit rate and volume exceed the implicit or explicit overhead. Let’s model the break-even for each provider using current public pricing (subject to change — verify before committing).

OpenAI GPT-4o (input: $2.50/MTok, cached: $1.25/MTok):

• No storage fee, no minimum request volume
• Break-even: any cache hit on ≥1,024 tokens saves $1.25/MTok
• Risk: cache misses on prefix changes cost you the full $2.50/MTok with no discount

Anthropic Claude 3.5 Sonnet (input: $3.00/MTok, cached: $0.30/MTok):

• 90% discount on ephemeral cache (5-min TTL)
• Minimum 1,024 cached tokens
• Break-even: one cache hit per 5-minute window on ≥1,024 tokens
• Persistent cache: 50% discount ($1.50/MTok) + storage fee

Google Gemini 1.5 Pro (input: $1.25/MTok, cached generation: $0.3125/MTok, storage: $0.025/MTok/hr):

• Storage fee applies regardless of requests
• Break-even volume: ~50 requests/hour on 1,024-token cache
• Below that threshold, you lose money vs. uncached requests

The Google model is uniquely sensitive to request density. A cache holding 10,000 tokens costs $0.25/hour. At 1,000 requests/hour with 10,000 cached tokens each, you save ~$9.37/hour on generation — a 37x return. At 10 requests/hour, you lose $0.22/hour.

Common pitfalls that erase savings

Dynamic content in the prefix

Timestamps, request IDs, user-specific data, or rotating API keys in the system prompt destroy cache eligibility. Move all dynamic content to the suffix or use template variables that resolve at inference time.

# Bad: cache key changes every request
system_prompt = f"You are a helpful assistant. Current time: {datetime.now()}"

# Good: static prefix, dynamic suffix
system_prompt = "You are a helpful assistant."
user_message = f"Current time: {datetime.now()}. User question: {question}"

Over-fragmenting the cache key

Including the full conversation history in the cache key means every new turn creates a new cache entry. For multi-turn conversations, cache only the system prompt and few-shots. Let the conversation history remain uncached — it’s typically small relative to the static prefix.

Ignoring minimum token thresholds

Sending 800 tokens of system prompt to OpenAI yields zero discount because the minimum is 1,024 tokens. Pad with harmless context (style guidelines, output format specs) to cross the threshold, or restructure to meet the minimum.

Cache stampedes on cold starts

When a popular cache expires, concurrent requests all miss and rebuild simultaneously. This spikes latency and costs. Implement probabilistic early refresh:

import random

async def maybe_refresh_cache(cache_key, ttl_remaining, rebuild_fn):
    # Refresh with increasing probability as TTL approaches zero
    refresh_probability = max(0, 1 - (ttl_remaining / 300))  # 5-min window
    if random.random() < refresh_probability:
        await rebuild_fn(cache_key)

Architectural patterns for reliable savings

Pattern 1: Immutable prompt templates

Version your prompt templates and treat them as immutable artifacts. Deploy new versions as new cache keys, not in-place edits. This gives you deterministic cache behavior and rollback capability.

prompts/
  v1/
    system.md
    fewshots.json
  v2/
    system.md
    fewshots.json

Hash the template directory to generate the cache key. A deployment that doesn’t change the hash preserves the cache.

Pattern 2: Cache-aware request batching

For batch workloads, sort requests by prompt prefix to maximize consecutive cache hits. This is especially effective with Anthropic’s explicit breakpoints and Google’s explicit caches.

def batch_by_cache_key(requests, cache_key_fn):
    """Group requests by cacheable prefix to maximize hit rate."""
    grouped = defaultdict(list)
    for req in requests:
        key = cache_key_fn(req)
        grouped[key].append(req)
    return grouped.values()

Pattern 3: Fallback with cache preservation

When your primary model is rate-limited or degraded, fail over to a secondary model but preserve the cache key structure so returning to primary recovers the discount.

async def generate_with_fallback(messages, primary="gpt-4o", secondary="claude-3.5-sonnet"):
    try:
        return await call_model(primary, messages)
    except RateLimitError:
        # Transform messages to secondary's cache format
        secondary_messages = adapt_for_secondary(messages)
        result = await call_model(secondary, secondary_messages)
        # Log cache key for primary so we can measure recovery
        await metrics.record_fallback(primary_cache_key=compute_key(messages))
        return result

The decisive takeaway

Prompt caching pricing discounts are real and substantial — 50-90% off input tokens for the cached portion — but they are not automatic. They require:

1. Structural discipline: Static prefixes ≥ provider minimums (1,024-2,048 tokens), dynamic content isolated in the suffix
2. Volume awareness: Google’s model demands sustained request density; OpenAI and Anthropic ephemeral caches reward bursty but recurring traffic
3. Routing coherence: Switching models or providers forfeits the cache; route with cache state as a first-class input
4. Observability: Track cache hit rate, prefix stability, and effective cost per request — not just headline token prices

Teams that treat prompt caching as an architectural concern — versioning templates, instrumenting hit rates, designing fallback paths that preserve cache eligibility — routinely achieve 30-60% effective input cost reduction. Teams that treat it as a transparent optimization see erratic savings and surprise bills.

The discount is not in the pricing page. It’s in the prompt architecture.