Choosing sample sizes for prompt A/B tests

Picking the right sample size for prompt A/B testing is the difference between trusting a fluke and shipping a real improvement. Too few evaluations and noise swamps the signal; too many and you burn tokens and latency budget on a foregone conclusion. This guide walks through the end-to-end calculation with runnable code.

Step 1: Define your metric and minimum detectable effect

Lock the evaluation metric before you write any experiment code. For most prompt changes the metric is either binary (correct answer, valid JSON, task completion within constraints) or continuous (LLM-judge score on a 1–5 scale, BLEU, latency). The metric type determines whether you reach for a z-test for proportions or a t-test for means.

Then set the minimum detectable effect (MDE). This is the smallest lift that justifies the engineering cost of a redeploy, a docs update, or a model swap. If your baseline task success is 70% and a prompt tweak moves it to 71%, that is usually not worth touching. Pick an MDE tied to product value: maybe 3 absolute points on success rate, or 0.2 points on a judge score.

The sample size for prompt A/B testing scales roughly with 1/MDE² for continuous metrics and similarly for proportions near 0.5. Halve the MDE and you quadruple the required traffic. Be honest about what your stakeholders will act on.

Step 2: Estimate baseline performance and variance

You cannot run a valid power calculation without a variance estimate. Pull a pilot batch of 50–100 examples through both the control prompt and the candidate. For a binary metric, record the success proportion p. For continuous, record mean and standard deviation.

import statistics

# pilot: 0/1 for binary task success
control_pilot = [1,0,1,1,0,1,1,1,0,1,1,0,1,1,1,0,1,1,1,0]
candidate_pilot = [1,1,1,1,0,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1]

p_control = sum(control_pilot) / len(control_pilot)
print(f"pilot control success: {p_control:.2f}")

# continuous judge scores (1-5)
scores = [4.1, 3.8, 4.5, 2.9, 4.0, 3.7, 4.2, 4.8, 3.5, 4.1]
mean = statistics.mean(scores)
std = statistics.stdev(scores)
print(f"pilot mean={mean:.2f} std={std:.2f}")

The pilot also surfaces labeling errors, ambiguous cases, and prompt-cache misses before you commit to a full run. If the pilot variance is far from your assumed value, revise the plan. A pilot that costs a few hundred tokens beats a full run that wastes tens of thousands on a mispowered test.

Step 3: Compute required sample size per variant

Use a two-sided test with α=0.05 and power=0.8 unless you have a domain-specific reason to deviate. For binary metrics, compute Cohen’s h effect size and solve for N per arm.

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

p1, p2 = 0.70, 0.75  # baseline and target after MDE
es = proportion_effectsize(p2, p1)
analysis = NormalIndPower()
n_per_arm = analysis.solve_power(
    effect_size=es,
    alpha=0.05,
    power=0.8,
    ratio=1,
    alternative='two-sided'
)
print(f"Required per variant: {n_per_arm:.0f}")

For 70% → 75% this prints 349. That is the sample size for prompt A/B testing per arm. For continuous metrics, use the pilot std dev directly:

from statsmodels.stats.power import TTestIndPower

std = 0.6
mde = 0.2
es = mde / std
n_per_arm = TTestIndPower().solve_power(
    effect_size=es,
    alpha=0.05,
    power=0.8,
    ratio=1,
    alternative='two-sided'
)
print(f"Required per variant: {n_per_arm:.0f}")

If your pilot showed higher variance than expected, the number goes up. Recompute before launching; do not trust a spreadsheet screenshot from last quarter.

Step 4: Adjust for multiple variants and guardrails

Testing three candidates against one control means four comparisons. Naive α=0.05 on each gives a family-wise error rate of about 18.5% (1−0.95⁴). Apply Bonferroni by dividing α by the number of comparisons, or use Holm–Bonferroni for slightly more power.

n_variants = 4
alpha_adjusted = 0.05 / n_variants
# feed alpha_adjusted into solve_power from Step 3

If you plan to peek at results daily, use sequential testing (AlphaInvesting, SPRT, or group-sequential boundaries). The simplest correct approach is still: fix the sample size up front, adjust α, and do not stop early without a proper sequential plan. Early stopping on a whim inflates false positives.

Step 5: Instrument the experiment and collect data

Assignment must be deterministic and logged. Hash a stable request identifier to split traffic, not a per-call random—you want a cohort that stays in its arm across retries.

import hashlib

def assign_variant(req_id: str, holdout=50) -> str:
    h = int(hashlib.md5(req_id.encode()).hexdigest(), 16) % 100
    return "control" if h < holdout else "candidate"

Log every response with a strict schema:

{
  "req_id": "abc123",
  "variant": "candidate",
  "timestamp": "2025-04-12T15:04:05Z",
  "latency_ms": 820,
  "prompt_tokens": 140,
  "completion_tokens": 32,
  "metric_binary": 1,
  "metric_score": 4.3
}

When scaling collection across providers, route through n4n.ai’s OpenAI-compatible endpoint to get automatic fallback and per-token metering so a provider outage doesn’t bias your sample size for prompt A/B testing. The gateway forwards cache-control hints, keeping prompt-cache hits consistent across arms and preventing a hidden confound where one arm pays more for context.

Collect over several days, not minutes, to avoid time-of-day or weekday bias in user inputs.

Step 6: Verify success and stop criteria

After collecting the precomputed N per arm, run the test. For binary outcomes:

from statsmodels.stats.proportion import proportions_ztest
import numpy as np

control_hits, control_n = 245, 349
candidate_hits, candidate_n = 280, 349
stat, pval = proportions_ztest(
    [candidate_hits, control_hits],
    [candidate_n, control_n]
)
# 95% CI on difference
diff = candidate_hits/candidate_n - control_hits/control_n
se = np.sqrt((p*(1-p)/n for p,n in [(0.75,349),(0.70,349)]).__next__())  # simplified
print(f"p={pval:.3f}, diff={diff:.3f}")

For continuous scores use scipy.stats.ttest_ind and compute the CI from the t-distribution.

Verification checklist:

• Collected N matches plan within ±2%. Dropouts happen; do not silently underpower.
• p-value is below your adjusted α from Step 4.
• The confidence interval on the difference excludes zero and its lower bound exceeds your MDE.
• Arm balance is close to 50/50 (or planned ratio) with no assignment leakage.
• No shift in latency or token count that indicates a confound rather than a prompt effect.

If those hold, you have a real result. If p > α, report the effect size estimate and its CI; do not dress up a non-significant result as a “trend” and ship it.

Reuse the calculation as code

The sample size for prompt A/B testing is not a one-time number. Every time you change the model, the judge, or the user population, rerun the power script. Check it into the repo next to the prompt templates so the next engineer sees the assumption chain: metric → MDE → pilot variance → N. That beats a Slack message saying “we tested 500 things.”