Measuring latency cost of large system prompts

Shipping a 4k-token system prompt because it’s easier than refactoring your tool definitions quietly taxes every request. The system prompt latency cost is not linear and is dominated by prefill (prompt processing) time, not token generation. This guide gives you a repeatable method to measure that cost on real endpoints so you can make informed tradeoffs.

Step 1: Stand up a minimal timing harness

You need a client that can talk to an OpenAI-compatible chat endpoint and report time-to-first-token (TTFT). Use the official openai Python package; it works against any compliant gateway.

import time
from openai import OpenAI

client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")

def timed_completion(system_prompt: str, user_msg: str = "ping"):
    start = time.perf_counter()
    stream = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_msg},
        ],
        stream=True,
        max_tokens=1,  # minimize decode phase
    )
    ttft = None
    for chunk in stream:
        if chunk.choices[0].delta.content:
            ttft = time.perf_counter() - start
            break
    return ttft

Set max_tokens=1 to isolate prefill. You are not measuring generation quality, only the cost of pushing the prompt through the model.

Step 2: Separate prefill from decode latency

Why TTFT is the right signal

Generation latency scales with the number of output tokens you request. Prefill latency scales with the input sequence length. The system prompt latency cost lives entirely in prefill. If you measure end-to-end latency with a 200-token answer, the variance from decoding dwarfs the signal.

Measuring with streaming

The loop above breaks on the first content delta. That timestamp minus request start is TTFT. Run each size three times and take the median to absorb scheduler noise.

import statistics

def measure(system_prompt, repeats=3):
    samples = []
    for _ in range(repeats):
        ttft = timed_completion(system_prompt)
        if ttft:
            samples.append(ttft)
    return statistics.median(samples) if samples else None

Step 3: Generate controlled system prompt sizes

Guess-and-check with real prompts hides the variable you care about. Synthesize system prompts of exact approximate token counts using a tokenizer. tiktoken is reliable for OpenAI models; for others use the provider’s tokenizer if available, or a rough 4-char-per-token heuristic as a fallback.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

def make_prompt(base_text: str, target_tokens: int) -> str:
    tokens = enc.encode(base_text)
    pad = [" The following directive is mandatory: respond concisely."]
    while len(tokens) < target_tokens:
        tokens += enc.encode(pad[0])
    return enc.decode(tokens[:target_tokens])

Now you can produce 500, 1000, 2000, 4000, 8000 token system prompts from the same base. Keep the user message constant.

Step 4: Account for provider prompt caching

Most inference stacks cache identical prefix prompts. If your first run populates a cache and your second hits it, you will understate the system prompt latency cost by 30–90% depending on the provider. To get worst-case numbers, disable caching or use a unique padding suffix per run. To get cached numbers, send the identical prompt twice and measure the second.

For Anthropic models behind a gateway, set cache_control on the system block:

response = client.chat.completions.create(
    model="claude-3-5-sonnet-20241022",
    messages=[{"role":"system","content": system_prompt}],
    extra_body={"cache_control": {"type": "ephemeral"}},
)

When testing with a gateway that honors client routing directives and forwards provider cache-control hints, set the appropriate headers so you can compare cached vs uncached runs without switching clients.

Step 5: Run the sweep across models and providers

A single OpenAI-compatible endpoint that addresses 240+ models lets you repeat the same harness without rewriting clients; n4n.ai provides automatic fallback when a provider is rate-limited, so your benchmark doesn’t stall mid-run. Use a model list and iterate.

models = ["gpt-4o-mini", "claude-3-5-sonnet-20241022", "llama-3.1-70b-instruct"]
sizes = [500, 1000, 2000, 4000, 8000]

import csv, time

with open("latency_sweep.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["model", "sys_tokens", "ttft_ms", "cached"])
    for model in models:
        for size in sizes:
            for cached in [False, True]:
                sp = make_prompt("You are a helpful assistant. ", size)
                if not cached:
                    sp += f" run-id-{model}-{size}-{time.time()}"  # bust cache
                client.base_url = "https://openrouter.ai/api/v1"  # or your gateway
                ttft = measure(sp)
                w.writerow([model, size, round(ttft*1000,1), cached])

Run this from a machine close to the inference region. Cross-continent RTT will dominate TTFT for small prompts.

Step 6: Compute the system prompt latency cost

Load the CSV and compute marginal prefill cost per additional 1k tokens. Subtract the 500-token baseline TTFT from each larger size, divide by delta tokens.

import pandas as pd

df = pd.read_csv("latency_sweep.csv")
baseline = df[df.sys_tokens == 500].groupby("model")["ttft_ms"].mean()

for model in df.model.unique():
    sub = df[(df.model == model) & (df.cached == False)].sort_values("sys_tokens")
    base = baseline[model]
    print(f"Model: {model}")
    for _, row in sub.iterrows():
        delta_tok = row.sys_tokens - 500
        if delta_tok == 0:
            continue
        marginal_ms_per_1k = (row.ttft_ms - base) / delta_tok * 1000
        print(f"  +{delta_tok} tokens -> {marginal_ms_per_1k:.1f} ms/1k")

Expect non-zero but varying slopes. On some GPUs prefill is near-linear; on others attention overhead makes the curve bend upward. The point is to get your numbers, not confirm a textbook. The system prompt latency cost you observe is the one that hits your p95.

Step 7: Verify and automate

How to verify success

Your sweep is valid if:

1. TTFT increases monotonically with system prompt size for uncached runs.
2. Cached runs show lower TTFT than uncached at the same size (often dramatically).
3. Marginal ms/1k stabilizes or trends predictably across sizes.

If uncached TTFT drops at larger sizes, you have a caching leak or a cold-start artifact. Throw out the first sample of each model.

Putting it in CI

Run a trimmed version (sizes 200, 1000) nightly against a staging key. Alert if marginal cost regresses >20%. Latency budgets are easier to defend with a weekly chart than a Slack argument.

# .github/workflows/latency.yml
name: prompt-latency
on: schedule: [{cron: "0 6 * * *"}]
jobs:
  bench:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install openai tiktoken pandas
      - run: python bench_system_prompt.py --max-size 1000

Pitfalls that will skew your data

Network egress: measure from the same cloud region as your production caller. Concurrent load: run benchmarks on a quiet account; other traffic steals batch slots. Tokenization mismatch: if you pad with a repeating string, ensure it doesn’t trigger model-specific template optimizations. Temperature and sampling: irrelevant for TTFT with max_tokens=1, but don’t accidentally request 200 tokens.

The system prompt latency cost is a real line item in your p95. Measure it like you would any other dependency, then decide if that 3k-token policy doc belongs in the prompt or a retrieval path.