A/B testing prompts for AI agents separates guesswork from measurable improvement when you ship changes to a production system. This post lays out an end-to-end pipeline: define a metric, version prompts, assign traffic deterministically, route with context, log everything, and analyze with a significance test.
Step 1: Define the decision metric before touching prompts
Pick a metric that reflects what the agent is actually for. If the agent books meetings, count confirmed bookings. If it answers support tickets, use a graded correctness score from a held-out evaluator or a downstream user thumbs-up. Do not optimize raw token count alone unless cost is the only lever.
Make the metric deterministic where possible
A human rating is noisy; a programmatic check is reproducible. For a SQL-generating agent, execute the query against a fixture and compare rows.
def evaluate_response(response: str, expected_rows: list) -> int:
try:
got = run_query(response)
return 1 if got == expected_rows else 0
except Exception:
return 0Log the raw score, not just the aggregate. You will need the distribution later.
Step 2: Create prompt variants with isolated configuration
Store prompts as versioned strings in a config file, not inline in your handler. This keeps the experiment swappable and reviewable in diffs.
{
"experiments": {
"onboarding_agent": {
"v1": "You are a concise onboarding assistant. Answer in one sentence.",
"v2": "You are an onboarding assistant. Use a numbered list and ask one follow-up."
}
}
}Load this at startup. Never mutate the live prompt in the request path without a version key attached.
Step 3: Implement consistent variant assignment
Random per-request assignment leaks across retries and makes debugging painful. Hash a stable identifier (user ID or session ID) so a given user always sees the same variant.
import hashlib
def assign_variant(user_id: str, exp_name: str, variants: list) -> str:
seed = f"{exp_name}:{user_id}".encode()
h = int(hashlib.sha256(seed).hexdigest(), 16)
return variants[h % len(variants)]
# variants = ["v1", "v2"]
# variant = assign_variant("user_123", "onboarding_agent", variants)Keep the bucket count small. Two or three variants is enough for a prompt change; more splits thin your sample size.
Step 4: Route requests with experiment context to the model
Pass the variant label in the request metadata so your logs and gateway can correlate traffic. If you front your models with an OpenAI-compatible gateway such as n4n.ai, you can pass a routing directive header and the gateway will honor it, forwarding cache-control hints to the provider to avoid recomputing identical prefixes across variants.
import requests
def call_model(prompt: str, variant: str, user_msg: str, api_key: str):
resp = requests.post(
"https://api.n4n.ai/v1/chat/completions",
headers={
"Authorization": f"Bearer {api_key}",
"x-route-directive": variant,
"x-cache-control": "ephemeral"
},
json={
"model": "openai/gpt-4o-mini",
"messages": [
{"role": "system", "content": prompt},
{"role": "user", "content": user_msg}
]
},
timeout=30
)
resp.raise_for_status()
return resp.json()The x-route-directive is a client routing hint; the gateway can use it to pin a provider or region. The cache-control hint tells the provider it can drop the prefix from long-term cache after the request.
Step 5: Log every interaction with full context
Write a single JSON line per interaction. Include the variant, the prompt text, the model response, the metric score, and a hashed user ID. Do not rely on centralized APM traces alone; you want a flat file you can pandas-read.
import json, time, hashlib
def log_interaction(path: str, user_id: str, variant: str, prompt: str,
response: dict, metric: int):
record = {
"ts": time.time(),
"uid": hashlib.sha256(user_id.encode()).hexdigest()[:16],
"variant": variant,
"prompt": prompt,
"completion": response["choices"][0]["message"]["content"],
"metric": metric
}
with open(path, "a") as f:
f.write(json.dumps(record) + "\n")Rotate the log daily. A 1 GB JSONL file is still manageable; a 50 GB one is not.
Step 6: Aggregate and analyze results
After a week of traffic (or enough volume to reach your precomputed sample size), load the logs and run a chi-square test on the success rate.
import pandas as pd
from scipy import stats
df = pd.read_json("interactions.jsonl", lines=True)
# metric is 1 for success, 0 for failure
ct = pd.crosstab(df["variant"], df["metric"])
chi2, p, _, _ = stats.chi2_contingency(ct)
print(ct)
print(f"chi2={chi2:.3f} p={p:.4f}")If p < 0.05 and the new variant’s success rate is higher, you have a winner. If p is large, you need more traffic or a bigger prompt delta. Do not ship on a 2% lift with 100 samples.
Watch the cost side
Compute average tokens per success. A prompt that lifts conversions but doubles cost may still fail ROI.
usage = pd.read_json("usage.jsonl", lines=True) # from gateway per-token metering
merged = df.merge(usage, on="uid")
cost_per_win = merged.groupby("variant")["tokens"].sum() / merged.groupby("variant")["metric"].sum()
print(cost_per_win)Step 7: Roll out the winner and clean up
Flip the default prompt version in your config. Keep the experiment code behind a feature flag for two weeks in case regressions appear in unseen slices of traffic.
DEFAULT_VARIANT = "v2" # was v1
def get_prompt(exp_name, user_id):
if flag_enabled("exp_onboarding"):
return PROMPTS[exp_name][assign_variant(user_id, exp_name, ["v1","v2"])]
return PROMPTS[exp_name][DEFAULT_VARIANT]Delete the variant assignment branch only after the flag is permanently on. Leaving dead experiment code is how future engineers get confused.
How to verify success
You have a working A/B test for prompts when:
- The metric delta is positive and the chi-square
p-valueis below your threshold (typically 0.05). - Latency p95 for the variant is within 10% of the control; a slower prompt that “feels” better will burn your SLA.
- Error rate (HTTP 5xx, parse failures) is statistically indistinguishable between buckets.
- Cost per successful interaction did not regress beyond the approved budget.
Run the analysis notebook weekly. If the variant wins consistently across two non-overlapping windows, promote it. A/B testing prompts for AI agents is not a one-shot; prompt drift and model updates mean the test you ran last month may not hold after a provider swaps a checkpoint. Keep the harness wired, even if you are not always running a live experiment.