Percentage-based rollouts for a new system prompt

Shipping a new system prompt to 100% of users on day one is how you quietly degrade conversion or triple your token bill before anyone notices. A percentage-based rollout for system prompt changes lets you expose the new instruction to a controlled slice of traffic, measure real behavior, and expand only when the data holds. Below is a deterministic, hash-based canary you can stand up in an afternoon with no specialized feature-flag vendor.

Step 1: Define prompt variants and rollout config

Start by extracting both the current and candidate system prompts into a versioned config. Keep them as raw strings; don’t interpolate user data into the system prompt at definition time.

{
  "rollout_pct": 5,
  "salt": "prod-prompt-2024-06",
  "variants": {
    "control": "You are a helpful assistant that answers clearly.",
    "treatment": "You are a helpful assistant. Answer in three sentences or fewer unless asked for detail."
  }
}

The rollout_pct is an integer 0–100. salt ensures you can re-bucket later by changing one value. Load this file at startup; hot-reload it so you can tune the percentage without a redeploy.

Step 2: Implement deterministic bucketing

Random assignment per request causes a user to see both prompts across refreshes, which poisons any per-user metric. Hash a stable identifier with the salt, take a modulo, and compare to the rollout percentage.

import hashlib

def in_treatment(user_id: str, salt: str, rollout_pct: int) -> bool:
    if not user_id:
        return False
    h = hashlib.sha256(f"{salt}:{user_id}".encode("utf-8")).hexdigest()
    bucket = int(h[:8], 16) % 100
    return bucket < rollout_pct

This is sticky: the same user_id always lands in the same bucket for a given salt and rollout_pct. If you must support anonymous traffic, use a persistent session cookie rather than IP—IP shifts and collapses buckets.

A percentage-based rollout for system prompt changes lives or dies on this function. Test it:

assert in_treatment("user_1", "s", 0) is False
assert in_treatment("user_1", "s", 100) is True

Step 3: Wire variant selection into the request path

Your LLM call should resolve the system prompt immediately before building the request. Never cache the prompt on the client beyond the request scope.

import json

with open("prompt_config.json") as f:
    cfg = json.load(f)

def resolve_system_prompt(user_id: str) -> str:
    if in_treatment(user_id, cfg["salt"], cfg["rollout_pct"]):
        return cfg["variants"]["treatment"]
    return cfg["variants"]["control"]

def chat(user_id: str, message: str):
    from openai import OpenAI
    client = OpenAI()  # or your OpenAI-compatible gateway
    sys_prompt = resolve_system_prompt(user_id)
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": sys_prompt},
            {"role": "user", "content": message},
        ],
    )
    return resp.choices[0].message.content

If you front calls with a gateway such as n4n.ai, it forwards provider cache-control hints, so you can tag the stable portion of your system prompt to retain prefix cache hits when only the experimental suffix changes. That avoids paying a caching penalty on every rollout bump.

Step 4: Emit structured logs and metrics

You cannot manage a percentage-based rollout for system prompt changes without per-variant telemetry. Log the variant, model, token counts, and latency on every response.

import logging, time

logger = logging.getLogger("prompt_rollout")

def chat_with_logging(user_id, message):
    start = time.monotonic()
    variant = "treatment" if in_treatment(user_id, cfg["salt"], cfg["rollout_pct"]) else "control"
    sys_prompt = cfg["variants"][variant]
    client = OpenAI()
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "system", "content": sys_prompt},
                  {"role": "user", "content": message}],
    )
    usage = resp.usage
    logger.info({
        "variant": variant,
        "user_id": user_id,
        "model": "gpt-4o-mini",
        "prompt_tokens": usage.prompt_tokens,
        "completion_tokens": usage.completion_tokens,
        "latency_ms": int((time.monotonic() - start) * 1000),
    })
    return resp.choices[0].message.content

Pipe these logs to your warehouse or metrics store. Create two counters: requests_total{variant} and tokens_total{variant}. A histogram on latency_ms per variant catches regressions early.

Step 5: Increase exposure gradually

Open the config and bump rollout_pct in steps: 1, 5, 20, 50, 100. Wait at least a few hours at each step if your traffic supports it. The point of a percentage-based rollout for system prompt changes is to let real usage surface problems that offline eval missed.

If you see completion token count climb 30% on the treatment variant with no quality gain, roll back to 0. Because bucketing is salted, you can also rotate salt to re-randomize if you suspect bias from a bad hash distribution.

Automate the bump with a cron or CI job that edits the JSON and reloads the service. Don’t manually SSH at 2am.

Step 6: Verify success and clean up

Define success before you start: e.g., “treatment holds or improves task-success rate measured by downstream rubric, and median completion tokens do not increase by more than 10%.” After reaching 100%, compare the last 24h of treatment against the control baseline from before the rollout.

Query your logs:

jq 'select(.variant=="treatment") | .completion_tokens' logs.jsonl | datamash median 1
jq 'select(.variant=="control") | .completion_tokens' logs.jsonl | datamash median 1

If treatment meets the bar, delete the control variant and the bucketing branch. Keep the salt in config for the next experiment.

How to verify the rollout is working now

During the rollout, sample live requests to confirm assignment matches the percentage. Hit your endpoint with 1,000 synthetic user IDs and count treatment assignments:

from collections import Counter
c = Counter(in_treatment(f"synthetic_{i}", cfg["salt"], cfg["rollout_pct"]) for i in range(1000))
print(c)  # approx {'False': 950, 'True': 50} at 5%

If the ratio is off by more than a couple percent, your hash or config load is broken. That check is the fastest way to trust the pipeline before real users feel it.

Stick to deterministic bucketing, emit per-variant usage, and let the numbers drive the percentage. Do that and a system prompt change becomes a boring, reversible config edit instead of a production incident.