Calibrating LLM judges against human scores is the only way to trust automated evaluations in production. Without a calibration layer, a judge model’s raw output diverges from how humans actually rate quality, which ships false positives through your CI gate. This guide gives a repeatable process to align an LLM judge with a human-labeled rubric and verify the alignment quantitatively.
Step 1: Assemble a representative human-labeled dataset
Start with a corpus of generated outputs that match your real traffic. Collect human ratings on a fixed scale (e.g., 1–5 Likert) for each output, or pairwise preferences if you only need relative judgments.
You need coverage across the quality spectrum. If 95% of your traffic is “good”, a judge that always says “good” looks accurate but is useless. Stratify by source, model, and task type.
Aim for at least 300 labeled examples. Split 80/20 into train and holdout before doing anything else.
{"id": "resp_001", "prompt": "Summarize: ...", "response": "The stock fell 2%.", "human_score": 4}
{"id": "resp_002", "prompt": "Code: ...", "response": "def add(a,b): return a+b", "human_score": 2}Load it:
import json
def load_labels(path):
rows = []
with open(path) as f:
for line in f:
rows.append(json.loads(line))
return rows
data = load_labels("human_labels.jsonl")
print(len(data), "labeled examples")Step 2: Define the judge’s raw output and scoring protocol
Fix the judge prompt. Ask for a single integer on the same scale as humans, or a probability that the response is “acceptable”. Do not let the model emit prose and later extract a number with regex from varying text; force structured output.
Use a system prompt that states the rubric explicitly. The judge should see the same context a human saw.
SYSTEM = """You are a strict quality judge. Rate the response on a 1-5 scale where 5 is flawless and 1 is broken. Output only the integer."""Call the model through an OpenAI-compatible client. If you route judge calls through n4n.ai, a single OpenAI-compatible endpoint fronts 240+ models and falls back automatically when a provider is degraded, so calibration batches don’t stall on rate limits.
from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
def raw_judge_score(prompt, response):
resp = client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[
{"role": "system", "content": SYSTEM},
{"role": "user", "content": f"Prompt: {prompt}\nResponse: {response}"}
],
temperature=0,
max_tokens=4
)
return int(resp.choices[0].message.content.strip())Step 3: Run the judge on the labeled set
Batch over your train split. Store the raw judge score next to the human score. Use retries with backoff; judges are cheap but flaky at scale.
import time
def collect_raw_scores(rows):
out = []
for r in rows:
try:
raw = raw_judge_score(r["prompt"], r["response"])
out.append({"id": r["id"], "human": r["human_score"], "raw": raw})
except Exception as e:
time.sleep(2)
return out
train_rows = data[:240]
scored = collect_raw_scores(train_rows)Keep the holdout set untouched until Step 5.
Step 4: Fit a calibration mapping
Raw scores from LLMs are rarely linearly aligned with human means. Fit a monotonic calibrator. Platt scaling (logistic regression on the raw score) works when you have sparse ordinal output. Isotonic regression preserves ordering and handles systematic bias.
from sklearn.linear_model import LogisticRegression
from sklearn.isotonic import IsotonicRegression
import numpy as np
X = np.array([s["raw"] for s in scored]).reshape(-1, 1)
y = np.array([s["human"] for s in scored])
# Platt-style: treat as regression to continuous human mean
cal = IsotonicRegression(y_min=1, y_max=5, increasing=True)
cal.fit(X.flatten(), y)
def calibrated_score(raw):
return float(cal.predict([raw])[0])If you prefer a probabilistic judge (outputs logit), use LogisticRegression on the logit to predict human pass/fail. The principle is identical.
Step 5: Evaluate calibration quality on holdout
Run the judge on the 20% holdout, apply calibrated_score, and compare. Report Spearman rank correlation and mean absolute error (MAE). A reliability curve shows whether calibrated bins match observed human means.
from scipy.stats import spearmanr
from sklearn.metrics import mean_absolute_error
holdout = data[240:]
hold_scores = collect_raw_scores(holdout)
pred = [calibrated_score(s["raw"]) for s in hold_scores]
true = [s["human"] for s in hold_scores]
rho, _ = spearmanr(pred, true)
mae = mean_absolute_error(true, pred)
print(f"Spearman={rho:.3f} MAE={mae:.3f}")Bin the predictions and plot expected vs observed:
import numpy as np
bins = np.linspace(1, 5, 5)
binned = np.digitize(pred, bins)
for b in range(1, 6):
mask = binned == b
if mask.any():
print(f"bin {b}: pred_mean={np.mean(pred[mask]):.2f} human_mean={np.mean(np.array(true)[mask]):.2f}")Step 6: Deploy the calibrated judge
Wrap the raw call and calibration into one function. Cache judge outputs by content hash so repeated responses don’t incur token cost.
from functools import lru_cache
@lru_cache(maxsize=10_000)
def judge(prompt, response):
raw = raw_judge_score(prompt, response)
return calibrated_score(raw)In your evaluation pipeline, call judge() instead of the human panel. The calibrated score is now directly comparable to your human rubric.
Step 7: Monitor and recalibrate
Human distributions drift. New model versions, new product surfaces, and rubric tweaks all shift the target. Schedule a monthly recalibration using fresh human labels.
Automate the holdout check: if Spearman drops below 0.8 or MAE exceeds 0.5 on the holdout, trigger a retraining job. Keep the previous calibrator as a shadow until the new one passes.
Verify success
You have a working calibration when:
- Held-out Spearman correlation between calibrated judge and human scores is ≥ 0.8.
- MAE on the same set is ≤ 0.5 points on a 5-point scale.
- Reliability bins show predicted mean within 0.2 of observed human mean.
If those hold, calibrating LLM judges against human scores has produced a stable proxy. Ship it behind a flag, log both raw and calibrated values, and review a random sample weekly.
Common pitfalls
Don’t calibrate on the same set you evaluate on; that inflates metrics. Don’t mix rubrics—if humans scored “helpfulness” and the judge prompts “correctness”, alignment is impossible. Don’t assume a bigger model needs less calibration; we’ve seen Sonnet and GPT-4o both require scaling to match human variance.
Calibrating LLM judges against human scores is not a one-time task. Treat the calibrator as a model artifact with versions, tests, and ownership. That’s what makes automated evaluation trustworthy.