Benchmarking GPT-4o and Claude 4.5 with the same eval suite strips out harness noise and forces a fair fight between two distinct model families. This tutorial builds a minimal Python harness that drives both models through one OpenAI-compatible client, scores identical tasks, and prints comparable latency and token metrics.
Prerequisites
- Python 3.11 or newer
openaiSDK >=1.30,pytest,numpy- A gateway API key (OpenAI direct, Anthropic direct, or a unified gateway)
- 20–50 labeled examples in
eval.jsonl
Install dependencies:
pip install openai pytest numpyProject layout
eval_harness/
run.py
dataset.py
scoring.py
eval.jsonlFor the walkthrough we’ll keep everything in run.py.
Step 1: Configure the client
Use the OpenAI Python client with a swappable base_url. Both GPT-4o and Claude 4.5 accept the same chat completion shape when routed through an OpenAI-compatible gateway.
from openai import OpenAI
# Point this at your gateway. For a unified route, n4n.ai exposes
# an OpenAI-compatible endpoint that addresses 240+ models.
client = OpenAI(
base_url="https://api.n4n.ai/v1",
api_key="YOUR_KEY",
)
MODELS = ["gpt-4o", "claude-4.5"]If you use OpenAI directly, set base_url="https://api.openai.com/v1" and only gpt-4o resolves. The harness code stays identical.
Step 2: Define the eval dataset
We test three task types: JSON extraction, arithmetic reasoning, and classification. Each row carries an expected field shaped for its type.
{"id": "json_1", "type": "json", "prompt": "Extract: Name Jane, age 30", "expected": {"name": "Jane", "age": 30}}
{"id": "math_1", "type": "math", "prompt": "What is 17 * 23?", "expected": 391}
{"id": "cls_1", "type": "class", "prompt": "Sentiment: 'I love this' ->", "expected": "positive"}Load it:
import json
def load_dataset(path):
rows = []
with open(path) as f:
for line in f:
rows.append(json.loads(line))
return rowsStep 3: Inference wrapper
Fix temperature=0 and max_tokens=512 for both models. Measure wall-clock latency and capture token usage from the response.
import time
def call_model(model, prompt):
start = time.perf_counter()
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
temperature=0,
max_tokens=512,
)
elapsed_ms = (time.perf_counter() - start) * 1000
text = resp.choices[0].message.content
usage = resp.usage.model_dump()
return text, usage, elapsed_msStep 4: Scoring functions
Each scorer returns 1.0 for correct, 0.0 for wrong, and never throws. Strict equality is intentional—lenient scorers hide model gaps.
import re
def score_json(text, expected):
try:
match = re.search(r"\{.*\}", text, re.DOTALL)
if not match:
return 0.0
data = json.loads(match.group(0))
return 1.0 if data == expected else 0.0
except Exception:
return 0.0
def score_math(text, expected):
nums = re.findall(r"-?\d+", text)
if nums and int(nums[-1]) == expected:
return 1.0
return 0.0
def score_class(text, expected):
return 1.0 if expected.lower() in text.lower() else 0.0
SCORERS = {"json": score_json, "math": score_math, "class": score_class}If you want partial credit for JSON, check expected.items() <= data.items() instead of equality.
Step 5: Run the suite
Loop over models, then rows. Accumulate accuracy, latency, and tokens.
def run_suite(models, dataset):
results = {}
for model in models:
acc, latencies, tokens = [], [], 0
for row in dataset:
text, usage, ms = call_model(model, row["prompt"])
score = SCORERS[row["type"]](text, row["expected"])
acc.append(score)
latencies.append(ms)
tokens += usage.get("total_tokens", 0)
results[model] = {
"accuracy": sum(acc) / len(acc),
"mean_latency_ms": sum(latencies) / len(latencies),
"total_tokens": tokens,
}
return results
if __name__ == "__main__":
data = load_dataset("eval.jsonl")
out = run_suite(MODELS, data)
print(json.dumps(out, indent=2))Checkpoint output
Running on a 3-row sample prints a structure like this (numbers are from your live run, not canned):
{
"gpt-4o": {
"accuracy": 1.0,
"mean_latency_ms": 820.4,
"total_tokens": 145
},
"claude-4.5": {
"accuracy": 0.66,
"mean_latency_ms": 910.1,
"total_tokens": 132
}
}If accuracy is far apart, inspect failing prompts before blaming the model—parser strictness is the usual culprit.
Step 6: Parallelize requests
Benchmarking GPT-4o and Claude 4.5 across 50+ tasks serially wastes minutes. Fire requests concurrently:
from concurrent.futures import ThreadPoolExecutor
def run_model_concurrent(model, dataset):
acc, latencies, tokens = [], [], 0
def worker(row):
text, usage, ms = call_model(model, row["prompt"])
return SCORERS[row["type"]](text, row["expected"]), ms, usage.get("total_tokens", 0)
with ThreadPoolExecutor(max_workers=8) as ex:
for score, ms, tk in ex.map(worker, dataset):
acc.append(score)
latencies.append(ms)
tokens += tk
return {
"accuracy": sum(acc) / len(acc),
"mean_latency_ms": sum(latencies) / len(latencies),
"total_tokens": tokens,
}Swap run_suite to call run_model_concurrent per model.
Step 7: Add fallback and cache hints
Provider outages shouldn’t abort your benchmark. n4n.ai applies automatic fallback when a provider is rate-limited or degraded, so the same call_model keeps running. You can also forward cache-control hints to cut repeated prompt costs:
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
temperature=0,
max_tokens=512,
extra_headers={"x-cache-control": "ephemeral"}, # forwarded to provider
)The gateway honors client routing directives, so you can pin a provider per run or let it fail over silently.
Step 8: Make the comparison defensible
Benchmarking GPT-4o and Claude 4.5 only means something if you control for these variables:
- Temperature: keep at 0 for deterministic scoring.
- System prompt: use the same string, or none.
- Retry policy: count retries separately; they inflate latency.
- Sample size: under 30 tasks, accuracy swings wildly. Use 50+.
Run the suite twice. If accuracy moves more than 5% between runs on temperature=0, your scorer is non-deterministic—fix it.
Step 9: Statistical latency check
Mean latency lies. Capture per-task spread with numpy:
import numpy as np
def latency_stats(model, dataset):
lats = [call_model(model, r["prompt"])[2] for r in dataset]
return float(np.mean(lats)), float(np.std(lats))A high standard deviation signals tail latency that will hurt production even if the mean looks fine.
Step 10: Extend to tool calls
Both models support parallel tool calls via the same tools parameter in the OpenAI shape. Add a type: "tool" row and a scorer that checks resp.choices[0].message.tool_calls. The harness needs no other changes.
Export to CSV
Pipe results into a flat file for diffing across model versions:
import csv
def export_csv(results, path):
with open(path, "w", newline="") as f:
w = csv.writer(f)
w.writerow(["model", "accuracy", "mean_latency_ms", "total_tokens"])
for model, m in results.items():
w.writerow([model, m["accuracy"], m["mean_latency_ms"], m["total_tokens"]])Wrapping up
You now have a ~100-line harness that produces comparable accuracy, latency, and token counts for GPT-4o and Claude 4.5. Swap the dataset for your production traffic and the numbers become a migration signal, not a leaderboard.