How caching affects token usage and cost reporting

The prompt caching effect on token usage is subtle: a cached prefix still appears in your request payload, but it stops consuming compute tokens and instead bills as cache-read tokens at a discount. Teams that aggregate prompt tokens without separating cache state will report inflated usage and misallocated cost. Accurate cost monitoring requires parsing provider usage objects at the token level.

How providers expose cached token counts

APIs do not hide caching, but they report it inconsistently. Anthropic’s Messages API returns cache_creation_input_tokens and cache_read_input_tokens at the usage level. OpenAI’s Chat Completions includes prompt_tokens_details.cached_tokens for models that support prompt caching.

{
  "usage": {
    "prompt_tokens": 8200,
    "completion_tokens": 300,
    "prompt_tokens_details": {
      "cached_tokens": 8000
    }
  }
}

The prompt_tokens total includes cached tokens. If you multiply prompt_tokens by your standard input price, you overpay in your model by 8000 tokens.

Anthropic’s shape is different:

{
  "usage": {
    "input_tokens": 8200,
    "output_tokens": 300,
    "cache_creation_input_tokens": 8000,
    "cache_read_input_tokens": 0
  }
}

Here input_tokens is the full prefix sent; the cache fields tell you how many of those were written or read.

Why naive aggregation distorts reports

Consider a service that sends an 8k-token system prompt prefix on every request. First call creates the cache; subsequent calls read it. A simple logger:

def log_usage(resp):
    u = resp["usage"]
    total_prompt = u.get("prompt_tokens", u.get("input_tokens", 0))
    store.add(total_prompt, u.get("completion_tokens", 0))

Run this for 100 requests. The first writes 8k cache tokens, the next 99 read them. Your store shows 100 * 8k = 800k prompt tokens billed at full input rate. Reality: 8k write (often at premium) + 99 * 8k read (at discount). Using Anthropic public rates—base input $3/MTok, cache write 1.25x, cache read 0.1x—naive cost is 800k * $3 = $2.40. Actual: write 8k * $3.75 = $0.03, reads 792k * $0.30 = $0.2376, total $0.2676. That is a 9x mismatch.

The prompt caching effect on token usage means your dashboards must split these categories or finance will question the bill.

Parsing cache-aware usage correctly

Write a normalizer that maps any provider response into a common schema. This keeps downstream cost logic simple.

def normalize_usage(resp):
    u = resp["usage"]
    if "prompt_tokens_details" in u:  # OpenAI-style
        cached = u["prompt_tokens_details"].get("cached_tokens", 0)
        prompt = u["prompt_tokens"]
        return {
            "cache_write": 0,
            "cache_read": cached,
            "compute_input": prompt - cached,
            "output": u["completion_tokens"],
        }
    # Anthropic-style
    write = u.get("cache_creation_input_tokens", 0)
    read = u.get("cache_read_input_tokens", 0)
    input_total = u.get("input_tokens", 0)
    return {
        "cache_write": write,
        "cache_read": read,
        "compute_input": input_total - write - read,
        "output": u["output_tokens"],
    }

Now you can meter each bucket independently.

Cost modeling with cache discounts

Public pricing: Anthropic charges 1.25x base for cache creation and 0.1x for cache read on Claude models. OpenAI applies a 50% discount to cached input on supported models. Your cost function must reference a rate table keyed by model and token type.

RATES = {
    "claude-3-5-sonnet": {"input": 3.0, "output": 15.0, "cache_write": 3.75, "cache_read": 0.30},
    "gpt-4o": {"input": 2.5, "output": 10.0, "cache_write": 2.5, "cache_read": 1.25},
}

def compute_cost(model, norm):
    r = RATES[model]
    return (norm["compute_input"] * r["input"] +
            norm["cache_write"] * r["cache_write"] +
            norm["cache_read"] * r["cache_read"] +
            norm["output"] * r["output"]) / 1_000_000

Rates are per million tokens; adjust to your actual contract. The point is separation.

Gateway behavior and cache-control hints

When you route through an inference gateway, cache directives must pass untouched. n4n.ai honors client routing directives and forwards provider cache-control hints, so a cache_control block in your Anthropic request reaches the upstream provider and the usage response reflects it. Your metering layer still receives provider-native usage; the gateway does not abstract cache token counts.

If you use a single OpenAI-compatible endpoint that addresses 240+ models, the response shape varies. Code a discriminator on usage keys as shown above.

Tradeoffs of caching for cost reporting

Caching saves money but adds observability burden:

• TTL volatility: Cache entries expire (often 5–60 minutes). A quiet period followed by a burst shows cache-write spikes that look like usage anomalies.
• Prefix rigidity: Changing one token in the cached prefix invalidates it. A/B tests that tweak system prompts silently shift cost from read to write.
• Multi-tenant isolation: If you share a prefix across customers, cache hits cross tenant boundaries. Your per-customer cost allocation must not credit one tenant for another’s warm cache.
• Provider drift: New models report cached tokens differently. Your normalizer needs maintenance.

The prompt caching effect on token usage is not just a discount; it is a different cost topology.

Building a cache-aware metering pipeline

1. Capture raw API responses, not just aggregated counts.
2. Normalize to {compute_input, cache_write, cache_read, output} per call.
3. Tag with model, provider, route, and tenant.
4. Store both raw and normalized rows; raw helps debug provider changes.
5. Compute cost in a batch job with a versioned rate table.
6. Alert on cache_read / (cache_read + cache_write + compute_input) dropping below expected thresholds.

Example query (SQL pseudo):

SELECT tenant_id,
       SUM(cache_read) AS read_tokens,
       SUM(cache_write) AS write_tokens,
       SUM(compute_input) AS compute_tokens
FROM usage_events
WHERE day = '2025-03-01'
GROUP BY tenant_id;

Feed those sums to your rate table.

Debugging cache misses

A sudden cost spike is often a cache miss, not traffic growth. Log the cache hit ratio per prefix signature. If you hash the static prefix and store it alongside usage, you can detect when a deployment rotated a prompt and silently killed the cache.

import hashlib

def prefix_key(messages):
    static_part = messages[0]["content"]  # assume system prompt first
    return hashlib.sha256(static_part.encode()).hexdigest()[:16]

# in handler
key = prefix_key(req["messages"])
norm = normalize_usage(resp)
emit_metric(f"cache_ratio:{key}", norm["cache_read"] / (norm["cache_read"] + norm["cache_write"] + norm["compute_input"] + 1))

Decisive takeaway

Treat cached tokens as a first-class metric from the first line of instrumentation. The prompt caching effect on token usage will otherwise corrupt every cost report downstream, turning real 70% savings into apparent overspend. Parse the provider usage object, split write/read/compute, and apply per-type rates. Do that and your LLM billing becomes trustworthy.