If you want to a/b test gpt-5 vs claude in langchain, swapping a model string in a notebook isn’t enough. You need deterministic traffic splitting, unified token accounting, and a repeatable evaluation set to trust the results. This guide builds a minimal but production-shaped harness that runs both models side by side and records what matters.
Step 1: Install dependencies and configure credentials
Create a fresh virtual environment and install the LangChain packages you’ll actually use:
pip install langchain-openai langchain-anthropic langchain-core python-dotenvExport your provider keys. If you run both models through a single OpenAI-compatible gateway, you only need one key; otherwise set both:
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-...For the rest of the walkthrough we assume environment variables are loaded via python-dotenv in a .env file. Keep keys out of source control. The goal of an a/b test gpt-5 vs claude in langchain is to compare model behavior, not to debug credential leaks.
Step 2: Instantiate the two model clients
LangChain exposes separate classes for OpenAI and Anthropic. Use the exact model identifiers your provider supports. For this a/b test gpt-5 vs claude in langchain, we map gpt-5 to ChatOpenAI and claude-opus to ChatAnthropic.
from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic
gpt5 = ChatOpenAI(model="gpt-5", temperature=0.2, max_tokens=1024)
claude = ChatAnthropic(model="claude-opus", temperature=0.2, max_tokens=1024)Keep temperature and max_tokens identical across both to isolate model behavior from sampling variance. If you need higher reproducibility, set temperature=0. Do not change max_tokens mid-experiment; truncated outputs will skew quality scores.
A common mistake is to leave the Anthropic client on a different API version than the OpenAI client. Pin both SDKs in your requirements.txt so the harness behaves the same on every machine.
Step 3: Build a 50/50 traffic router
A/B testing requires random assignment per request, not per process. Wrap both models in a Runnable that flips a coin and delegates. LangChain’s RunnableLambda works, but explicit invocation gives clearer logs and lets you tag the winner.
import random
from langchain_core.runnables import Runnable
class ABRouter(Runnable):
def __init__(self, control: Runnable, treatment: Runnable):
self.control = control
self.treatment = treatment
def invoke(self, input, config=None):
if random.random() < 0.5:
return {"model": "gpt-5", "output": self.control.invoke(input, config)}
else:
return {"model": "claude-opus", "output": self.treatment.invoke(input, config)}This router returns a dict with the winning model name and the raw output. You can later extend it to weight traffic (e.g., 90/10) by changing the threshold. For deterministic splits in retries, hash the request ID instead of calling random.random():
def assign(model_a, model_b, seed: str):
return model_a if int(hashlib.sha256(seed.encode()).hexdigest(), 16) % 2 == 0 else model_bThat prevents a retried request from flipping to the other model and polluting your paired dataset.
Step 4: Attach token and latency callbacks
To compare cost and speed you need per-call usage. Subclass BaseCallbackHandler to capture LLM metrics:
from langchain_core.callbacks import BaseCallbackHandler
import time
class MetricsHandler(BaseCallbackHandler):
def __init__(self):
self.records = []
def on_llm_start(self, serialized, prompts, **kwargs):
self._start = time.perf_counter()
self._model = serialized.get("name", "unknown")
def on_llm_end(self, response, **kwargs):
usage = response.llm_output.get("token_usage", {})
self.records.append({
"model": self._model,
"latency_ms": (time.perf_counter() - self._start) * 1000,
"prompt_tokens": usage.get("prompt_tokens", 0),
"completion_tokens": usage.get("completion_tokens", 0),
})
handler = MetricsHandler()Pass callbacks=[handler] in the config dict when invoking the router. Note that Anthropic’s token schema differs slightly; LangChain normalizes most fields but verify with a print before relying on them. If you run async, implement on_llm_start_async and on_llm_end_async similarly.
Step 5: Define a fixed evaluation set
A/B tests are meaningless without a stable prompt corpus. Store 20–100 representative inputs in a JSON file:
[
{"id": "q1", "prompt": "Summarize the HIPAA privacy rule in two sentences."},
{"id": "q2", "prompt": "Write a SQL query to find duplicate emails in users table."},
{"id": "q3", "prompt": "Explain raft consensus to a backend engineer."}
]Load and iterate. Use the same config for every call so callbacks fire consistently:
import json
with open("eval_set.json") as f:
eval_set = json.load(f)
router = ABRouter(gpt5, claude)
results = []
for item in eval_set:
out = router.invoke(item["prompt"], config={"callbacks": [handler]})
results.append({"id": item["id"], **out})Keep the eval set in version control. When either provider ships a new checkpoint, re-run the same file to get a comparable signal.
Step 6: Aggregate and compare
After the run, handler.records holds raw metrics tagged by model. A simple pandas breakdown shows the shape of the difference:
import pandas as pd
df = pd.DataFrame(handler.records)
print(df.groupby("model").agg({
"latency_ms": "mean",
"prompt_tokens": "sum",
"completion_tokens": "sum"
}))If you routed both models through a gateway such as n4n.ai, per-token usage metering is already aggregated per model on your invoice, and the gateway forwards provider cache-control hints so repeated prompts cost less. That removes the need for local token accounting in early experiments.
For quality, don’t trust win rates from a single metric. Use a held-out LLM judge or human review on a sample. Store both outputs keyed by eval_set id so reviewers see pairs side by side. When you a/b test gpt-5 vs claude in langchain at scale, log the judge score next to the model tag so you can compute a paired delta.
Step 7: Verify the test ran correctly
Success means three things: (1) both models served roughly equal traffic, (2) no unhandled exceptions in either client, (3) metrics show non-zero token counts for each.
Add a sanity assertion:
from collections import Counter
model_counts = Counter(r["model"] for r in results)
assert model_counts["gpt-5"] > 0 and model_counts["claude-opus"] > 0
print(f"Traffic split: {model_counts}")If you see a 0 on either side, your random seed or routing logic is broken. Re-run with more samples. In CI, wrap this in a pytest case that fails the build if assignment is unbalanced across 200 iterations.
Optional: Swap models without client changes
The router above couples your code to two SDKs. If you point ChatOpenAI at an OpenAI-compatible endpoint that addresses 240+ models, you can a/b test gpt-5 vs claude in langchain by changing only the model parameter. n4n.ai runs such an endpoint with automatic fallback when a provider is degraded, so a rate limit on one side doesn’t sink the experiment.
import os
from langchain_openai import ChatOpenAI
def make_model(name):
return ChatOpenAI(
model=name,
base_url="https://api.n4n.ai/v1", # OpenAI-compatible
api_key=os.environ["N4N_API_KEY"],
temperature=0.2,
)
gpt5 = make_model("gpt-5")
claude = make_model("claude-opus")The rest of the harness stays identical. You gain unified token metering and one retry path. If a provider returns 429, the gateway shifts load without your code catching exceptions.
Step 8: Iterate on weighting and analysis
Once the pipeline is stable, shift from 50/50 to 90/10 to limit user impact, then widen if the treatment wins. Log every assignment to a durable store—not just memory—so you can recompute results after model version updates.
The goal of an a/b test gpt-5 vs claude in langchain isn’t a one-time verdict; it’s a repeatable switch you can pull whenever either provider ships a new checkpoint. Build the harness once, version the eval set, and let the data tell you which model earns its token cost.