How to structure prompts for better cache reuse

Prompt caching only works when the leading tokens of your request match a previous request exactly. Most teams leave money on the table because they vary system prompts, shuffle message order, or inject dynamic content at the wrong position. This guide shows how to structure prompts for cache reuse so the provider can serve the common prefix from memory instead of recomputing it.

The mechanics are straightforward: providers like Anthropic, OpenAI, and Google cache the longest matching prefix of your conversation history. If your first 2,000 tokens are identical to a prior request, those tokens are fetched from cache at a fraction of the cost and latency. The catch is that tiny differences — a timestamp, a user ID, a reordered few-shot example — break the match entirely. The steps below eliminate those differences.

Step 1: Pin the system prompt and static context at the very start

The system prompt and any reference material (docs, schemas, few-shot examples) must be the first tokens in every request. Put them in the same order every time. Do not interpolate variables into the system prompt; pass variable data in the user message instead.

# Bad: system prompt changes per request
system_prompt = f"You are a helpful assistant. Current user: {user_id}. Today is {date}."

# Good: static system prompt, dynamic data in user message
SYSTEM_PROMPT = """You are a helpful assistant.
Reference the user context provided in the user message."""

user_message = f"""User context:
- user_id: {user_id}
- date: {date}

Task: {task}"""

When you call the API, the message array should look like:

[
  {"role": "system", "content": "You are a helpful assistant.\nReference the user context provided in the user message."},
  {"role": "user", "content": "User context:\n- user_id: u_123\n- date: 2025-01-15\n\nTask: Summarize the attached document."}
]

Every request now shares the exact same system prompt tokens. The cache key starts at token 1 and extends through the entire system prompt.

Step 2: Keep few-shot examples in a fixed canonical order

Few-shot examples are high-value cache candidates because they’re long and static. Define them once as a constant list and never reorder, truncate, or modify them per request.

FEW_SHOT_EXAMPLES = [
    {"input": "Classify: 'I love this product!'", "output": "positive"},
    {"input": "Classify: 'This is broken.'", "output": "negative"},
    {"input": "Classify: 'It works okay.'", "output": "neutral"},
]

def build_classification_prompt(text: str) -> str:
    parts = ["Classify the sentiment of the following text.\n\nExamples:"]
    for ex in FEW_SHOT_EXAMPLES:
        parts.append(f"Input: {ex['input']}\nOutput: {ex['output']}")
    parts.append(f"\nInput: {text}\nOutput:")
    return "\n".join(parts)

If you need to select a subset of examples (e.g., for token budget), choose a deterministic policy: always take the first N, or hash the input and pick a consistent slice. Random sampling breaks caching.

Step 3: Put all dynamic content at the end of the user message

The user message should follow a template where the variable parts — user input, retrieved documents, tool results — appear after a fixed prefix. The fixed prefix becomes part of the cache key; the variable suffix does not.

USER_TEMPLATE = """Context:
{context}

Question:
{question}

Answer concisely."""

def render_user_message(context: str, question: str) -> str:
    return USER_TEMPLATE.format(context=context, question=question)

If you use retrieval-augmented generation, the retrieved chunks go into {context}. The template string itself — “Context:\n{context}\n\nQuestion:\n{question}\n\nAnswer concisely.” — is static and cacheable. Only the filled-in values vary.

Step 4: Normalize whitespace and formatting exactly

Whitespace differences break prefix matching. A trailing newline, an extra space, or a different indentation style produces a different token sequence. Enforce a single canonical format.

def normalize_prompt(text: str) -> str:
    # Strip trailing whitespace from each line, ensure single trailing newline
    lines = [line.rstrip() for line in text.splitlines()]
    # Remove leading/trailing blank lines
    while lines and not lines[0]:
        lines.pop(0)
    while lines and not lines[-1]:
        lines.pop()
    return "\n".join(lines) + "\n"

Apply this to your system prompt, few-shot blocks, and user template at startup. Store the normalized strings as constants. Do not normalize at request time — that adds latency and risks drift.

Step 5: Use a stable tokenizer-aware token counter for verification

You cannot rely on character counts. Providers cache at token granularity, and tokenizers vary by model. Use the provider’s tokenizer (or a compatible open-source equivalent) to measure the exact cached prefix length.

import tiktoken

# For OpenAI-compatible models
enc = tiktoken.encoding_for_model("gpt-4o")

def count_tokens(text: str) -> int:
    return len(enc.encode(text))

# Verify your static prefix length
STATIC_PREFIX = SYSTEM_PROMPT + "\n" + "\n".join(
    f"Input: {ex['input']}\nOutput: {ex['output']}" for ex in FEW_SHOT_EXAMPLES
) + "\n" + USER_TEMPLATE.split("{context}")[0]  # up to the first variable

print(f"Static prefix tokens: {count_tokens(STATIC_PREFIX)}")

Run this at build time or in a test. If the count changes, something in your static content drifted. Lock the tokenizer version in your dependencies.

Step 6: Send the same model identifier every time

Cache is scoped to the model. Requests to gpt-4o and gpt-4o-2024-08-06 are separate cache namespaces. Pin the exact model string in your configuration and never override it per request.

# config.py
MODEL = "gpt-4o-2024-08-06"  # explicit version, not "gpt-4o"

# client.py
from config import MODEL

def chat(messages: list[dict], **kwargs):
    return openai.chat.completions.create(
        model=MODEL,
        messages=messages,
        **kwargs
    )

If you route across providers (e.g., Anthropic as fallback), understand that each provider maintains its own cache. A request served by Anthropic does not populate OpenAI’s cache. Design your routing so that repeated workloads stick to one provider when possible.

Step 7: Verify cache hits in production

Providers expose cache metrics in response headers or usage fields. Capture and alert on them.

Anthropic returns cache_creation_input_tokens and cache_read_input_tokens in the usage object:

response = anthropic.messages.create(
    model="claude-3-5-sonnet-20241022",
    messages=messages,
)

usage = response.usage
cache_read = getattr(usage, "cache_read_input_tokens", 0)
cache_write = getattr(usage, "cache_creation_input_tokens", 0)
total_input = usage.input_tokens

hit_rate = cache_read / (cache_read + cache_write) if (cache_read + cache_write) > 0 else 0
print(f"Cache hit rate: {hit_rate:.1%} ({cache_read} read / {cache_write} write)")

OpenAI returns prompt_tokens_details.cached_tokens in the usage object (available on gpt-4o and later):

response = openai.chat.completions.create(
    model="gpt-4o-2024-08-06",
    messages=messages,
)

cached = response.usage.prompt_tokens_details.cached_tokens
total_prompt = response.usage.prompt_tokens
print(f"Cached tokens: {cached} / {total_prompt} ({cached/total_prompt:.1%})")

Log these metrics per request. Build a dashboard showing cache hit rate over time. A drop signals a prompt structure regression — often a new dynamic field injected at the wrong position.

Step 8: Guard against cache poisoning from user-controlled input

If user input appears early in the message array (e.g., a chat history where the user’s first message varies), the cache key diverges immediately. Restructure so user-controlled content starts after your static prefix.

# Chat history: system + static context + conversation
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": STATIC_CONTEXT_BLOCK},  # your docs, schemas, etc.
]

# Append conversation history — this part varies per session
for turn in conversation_history:
    messages.append({"role": turn.role, "content": turn.content})

# Current user message goes last
messages.append({"role": "user", "content": current_user_message})

The static prefix now includes system prompt + static context block. Only the conversation history and current message vary. For long-running sessions, consider summarizing old turns into a fixed “session summary” block that you update infrequently, keeping the varying tail short.

Step 9: Handle streaming and tool calls without breaking the prefix

Streaming does not affect caching — the cache key is determined at request start. Tool calls do affect it if the tool definition or prior tool results appear in the message array before the static prefix ends.

Define tools in the API call’s tools parameter, not in the message content. Keep tool results in the message history after your static prefix.

TOOLS = [
    {"type": "function", "function": {"name": "search", "parameters": {...}}},
]

# Request
response = openai.chat.completions.create(
    model=MODEL,
    messages=messages,  # static prefix + conversation history
    tools=TOOLS,        # tools defined here, not in messages
)

If you inject tool schemas into the system prompt for some reason, that schema becomes part of the static prefix — which is fine as long as it never changes.

Step 10: Automate regression tests for cache stability

Add a test that hashes the static prefix tokens and fails if the hash changes. This catches accidental edits to system prompts, few-shot examples, or templates.

# test_cache_stability.py
import hashlib
import tiktoken
from prompts import SYSTEM_PROMPT, FEW_SHOT_EXAMPLES, USER_TEMPLATE

enc = tiktoken.encoding_for_model("gpt-4o-2024-08-06")

def static_prefix_tokens() -> list[int]:
    parts = [SYSTEM_PROMPT]
    for ex in FEW_SHOT_EXAMPLES:
        parts.append(f"Input: {ex['input']}\nOutput: {ex['output']}")
    parts.append(USER_TEMPLATE.split("{context}")[0])
    full = "\n".join(parts) + "\n"
    return enc.encode(full)

def test_static_prefix_stable():
    tokens = static_prefix_tokens()
    token_hash = hashlib.sha256(str(tokens).encode()).hexdigest()
    # Store this expected hash in the test or a companion file
    EXPECTED_HASH = "a1b2c3d4e5f6..."  # update intentionally when you change prompts
    assert token_hash == EXPECTED_HASH, "Static prompt prefix changed — cache will miss"

Run this in CI. When you intentionally update prompts, update the expected hash in the same PR. This forces the team to acknowledge the cache impact of every prompt change.

Verification checklist

Before deploying, confirm each of these:

1. Static prefix token count matches your design target (e.g., 1,500–3,000 tokens for meaningful savings).
2. Cache hit rate in staging exceeds 80% for repeated workloads after warm-up.
3. Latency p99 drops by at least 30% on cache hits versus cold requests (provider-dependent, but directionally consistent).
4. Cost per request reflects the provider’s cached-token pricing (typically 10–50% of uncached input token price).
5. No dynamic content appears before the last static token in the message array — verify by tokenizing a sample request and inspecting the token stream.

Common failure patterns

What good looks like

A well-structured request sends 2,500 static tokens (system prompt + 20 few-shot examples + RAG template) followed by 200 dynamic tokens (retrieved chunks + user question). On the 100th request with the same static prefix, the provider reads 2,500 tokens from cache at ~10% cost and ~40% latency. The dynamic 200 tokens are computed fresh. Multiply by millions of requests — the savings compound.

Structure your prompts once. Verify with token counts and production metrics. Treat the static prefix as a contract: change it deliberately, measure the impact, and update your regression test.