How to A/B test GPT-5 against Claude Sonnet 4.5

When you need to decide which frontier model to ship, A/B testing GPT-5 vs Claude Sonnet 4.5 in production traffic is the only way to get ground truth on latency, quality, and cost. This guide walks through a concrete implementation using a single OpenAI-compatible endpoint and a simple assignment layer you control.

Step 1: Define metrics and traffic split before writing code

Before you start A/B testing GPT-5 vs Claude Sonnet 4.5, lock down the decision criteria. Pick one primary success metric (e.g., task completion rate, user thumbs-up, or parseable JSON rate) and at least two guardrail metrics (p95 latency, error rate). Token cost matters but varies with prompt length; log raw token counts and compute cost later with your negotiated rates.

Traffic split should be stable per user to avoid flicker. A 50/50 split gives the fastest statistical power for equal arms. If you are risk-averse, run a 90/10 canary first. Use a deterministic hash of user_id so the same user always hits the same model within an experiment window.

import hashlib

def assign_arm(user_id: str, salt: str, split_pct: int = 50) -> str:
    h = hashlib.sha256(f"{salt}:{user_id}".encode()).hexdigest()
    bucket = int(h[:8], 16) % 100
    return "gpt5" if bucket < split_pct else "sonnet"

Step 2: Establish unified model access

You do not want two SDKs, two auth flows, and two retry implementations. A gateway such as n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models, so you avoid juggling separate Anthropic and OpenAI clients. Point the standard openai package at the gateway and switch models by name.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",  # swap for your gateway URL
    api_key="sk-your-key",
)

MODEL_NAMES = {
    "gpt5": "gpt-5",
    "sonnet": "claude-sonnet-4.5",
}

def complete(arm: str, messages: list, **kwargs):
    model = MODEL_NAMES[arm]
    return client.chat.completions.create(
        model=model,
        messages=messages,
        **kwargs,
    )

If you use prompt caching, pass provider cache-control hints via extra_body. The gateway forwards them to the upstream provider without translation.

Step 3: Implement assignment and request routing

Wire the assignment function into your request handler. Keep the arm decision independent of the model call so you can log the intended arm even if the provider falls back.

import time, json, sys

def handle_chat(user_id: str, messages: list, exp_salt: str):
    arm = assign_arm(user_id, exp_salt)
    start = time.time()
    try:
        resp = complete(arm, messages, temperature=0.2)
        latency = (time.time() - start) * 1000
        log_event(arm, resp, latency)
        return resp.choices[0].message.content
    except Exception as e:
        log_error(arm, str(e))
        raise

This is the production path. Each user sees only one model, which is what makes it a true A/B test rather than a side-by-side eval.

Step 4: Log the right fields

The logs from A/B testing GPT-5 vs Claude Sonnet 4.5 must support both cost and quality analysis. Capture usage, latency, and the actual model served (in case of fallback). Write to JSONL for easy ingestion.

def log_event(arm: str, resp, latency_ms: float):
    record = {
        "ts": time.time(),
        "arm": arm,
        "served_model": resp.model,
        "prompt_tokens": resp.usage.prompt_tokens,
        "completion_tokens": resp.usage.completion_tokens,
        "latency_ms": latency_ms,
        "finish_reason": resp.choices[0].finish_reason,
    }
    sys.stderr.write(json.dumps(record) + "\n")

Redirect stderr to a file or pipe to your log collector. Do not log full prompts in production unless you have a retention policy; a hash is enough for deduplication.

Step 5: Run shadow calls for quality evaluation

Split traffic tells you about engagement and errors, but not whether responses are objectively better. For a small fraction of requests, call both models concurrently and store both outputs for offline judging.

from openai import AsyncOpenAI
import asyncio

aclient = AsyncOpenAI(base_url="https://api.n4n.ai/v1", api_key="sk-your-key")

async def shadow_call(messages):
    gpt_task = aclient.chat.completions.create(model="gpt-5", messages=messages)
    sonnet_task = aclient.chat.completions.create(model="claude-sonnet-4.5", messages=messages)
    gpt_resp, sonnet_resp = await asyncio.gather(gpt_task, sonnet_task)
    return gpt_resp.choices[0].message.content, sonnet_resp.choices[0].message.content

Queue these to a dataset and score with a rubric or an LLM judge. This isolates quality from the noise of live user behavior.

Step 6: Analyze with statistical rigor

After a week of A/B testing GPT-5 vs Claude Sonnet 4.5, load the JSONL and compute per-arm summaries. Use Pandas for aggregation and SciPy for significance.

import pandas as pd
from scipy import stats

df = pd.read_json("ab_logs.jsonl", lines=True)
summary = df.groupby("arm").agg(
    n=("arm", "size"),
    p50_lat=("latency_ms", lambda x: x.quantile(0.5)),
    p95_lat=("latency_ms", lambda x: x.quantile(0.95)),
    avg_prompt_tok=("prompt_tokens", "mean"),
    avg_comp_tok=("completion_tokens", "mean"),
)
print(summary)

# Example: latency difference
gpt_lat = df[df.arm == "gpt5"].latency_ms
son_lat = df[df.arm == "sonnet"].latency_ms
u_stat, p_val = stats.mannwhitneyu(gpt_lat, son_lat, alternative="two-sided")
print(f"latency p-value: {p_val:.4f}")

For binary success (e.g., finish_reason == "stop" and downstream parse succeeded), use a chi-square or two-proportion z-test. Do not declare a winner on a 2% latency dip with n=50; wait for p < 0.05 and a practically meaningful effect size.

Step 7: Verify the experiment executed as intended

Before trusting the numbers, confirm the test actually ran correctly:

1. Both arms appear in logs with counts near your split ratio (e.g., 48/52 is fine, 0/100 is not).
2. served_model matches the expected model for the arm. If your gateway does automatic fallback under rate limits, some calls may have silently shifted to a different model; those rows must be excluded or flagged.
3. No systematic error bias: check that error rates per arm are not skewed by a broken key or a misconfigured region.
4. Timestamps span the intended window; a deploy that restarted the service mid-experiment can truncate data.

A quick check:

assert set(df.arm.unique()) == {"gpt5", "sonnet"}
assert df.groupby("arm").size().min() > 100  # minimum sample
mismatch = df[df.apply(lambda r: (r.arm=="gpt5" and "gpt-5" not in r.served_model) or (r.arm=="sonnet" and "claude" not in r.served_model), axis=1)]
print(f"fallback rows: {len(mismatch)}")

Step 8: Promote the winner or iterate

If one arm wins on the primary metric without regressing guardrails, shift traffic gradually: 10% → 50% → 100%. Keep the assignment code in place so you can reverse the switch instantly by changing split_pct or exp_salt.

If results are inconclusive, extend the window or refine the prompt for the lagging model. Model behavior is sensitive to system prompts; a poorly tuned Claude Sonnet 4.5 prompt can lose to a well-tuned GPT-5 prompt for reasons unrelated to model quality.

A/B testing GPT-5 vs Claude Sonnet 4.5 is not a one-shot script. It is a measurement loop: define, route, log, judge, decide. The engineering effort is mostly in clean logging and stable assignment—the model calls themselves are the easy part.