Raw LLM judges that emit a score in one shot swing by several points between identical runs. Applying chain-of-thought prompting LLM judge techniques—forcing the model to write its reasoning before the numeric verdict—cuts that variance dramatically. This guide gives you a working pipeline you can drop into any evaluation harness.
Step 1: Define the evaluation task and scoring rubric
Consistency starts with an unambiguous target. If the judge does not know what “good” means, no prompt trick will save you. Pick a concrete task: grading factual accuracy of a support answer, scoring code correctness, or rating tone. Then write a rubric as a fixed JSON schema so every run evaluates the same dimensions.
{
"criteria": [
{"name": "factual_accuracy", "weight": 0.5, "scale": "0-2"},
{"name": "completeness", "weight": 0.3, "scale": "0-2"},
{"name": "tone", "weight": 0.2, "scale": "0-1"}
],
"final_scale": "1-5"
}Map the weighted sub-scores to your final scale in the prompt. Engineers often skip this and wonder why the judge oscillates; the model is inferring the rubric fresh each call.
Step 2: Design a chain-of-thought prompt template
The core of chain-of-thought prompting LLM judge work is a template that forbids skipping straight to the number. Instruct the model to reason, then score. Use a strict output contract so later parsing is trivial.
SYSTEM_PROMPT = """You are a strict evaluation judge.
Follow these steps:
1. Read the candidate answer and the reference.
2. Reason through each rubric criterion one by one.
3. Output ONLY a JSON object with keys:
"reasoning" (string, your step-by-step analysis),
"score" (integer on the final scale).
Do not write any text outside the JSON."""
USER_TEMPLATE = """Rubric: {rubric}
Reference: {reference}
Candidate: {candidate}
Judge according to the system instructions."""Why this reduces variance
A bare “score this 1-5” call lets the model collapse nuanced trade-offs into a gut feel. Forcing explicit reasoning anchors the final number to stated observations. The reasoning text also gives you a debug trail when the score looks wrong.
Step 3: Call the model with structured outputs
Use an OpenAI-compatible client. If you point the client at n4n.ai’s OpenAI-compatible endpoint, you get access to 240+ models and automatic fallback when a provider is rate-limited, so judge runs don’t silently fail mid-batch. Set temperature=0 for judges; randomness is the enemy of consistency.
from openai import OpenAI
client = OpenAI(
base_url="https://api.n4n.ai/v1",
api_key="YOUR_KEY"
)
def judge(reference: str, candidate: str, rubric: dict) -> dict:
resp = client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
temperature=0,
response_format={"type": "json_object"},
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": USER_TEMPLATE.format(
rubric=rubric, reference=reference, candidate=candidate)}
]
)
return json.loads(resp.choices[0].message.content)The response_format flag pushes the model toward valid JSON, but you still must validate.
Step 4: Parse and isolate the reasoning from the score
Never trust the raw string. Extract both fields and validate types. Keep the reasoning in your logs; it is the only way to audit a score.
import json
def safe_parse(raw: str) -> tuple[str, int]:
try:
obj = json.loads(raw)
reasoning = str(obj["reasoning"])
score = int(obj["score"])
return reasoning, score
except (json.JSONDecodeError, KeyError, ValueError) as e:
raise ValueError(f"Malformed judge output: {e}")If you batch thousands of items, wrap this in a retry that re-requests with a stricter reminder when parsing fails.
Step 5: Run multiple trials and measure consistency
A single call tells you nothing about stability. Run the same input N times and look at the spread. Chain-of-thought prompting LLM judge setups typically drop the standard deviation by half versus zero-shot scoring on the same model.
import numpy as np
def consistency_check(reference: str, candidate: str, rubric: dict, n: int = 5):
scores = []
for _ in range(n):
raw = judge(reference, candidate, rubric)
_, score = safe_parse(raw)
scores.append(score)
arr = np.array(scores)
return {
"mean": float(arr.mean()),
"std": float(arr.std()),
"scores": scores
}
result = consistency_check(ref, cand, rubric, n=7)
print(result["std"]) # target: < 0.5 on a 1-5 scaleStore the per-trial reasoning. If one trial diverges, read its reasoning—usually it reveals an ambiguous rubric phrase.
Step 6: Calibrate and prune the rubric
After running a few hundred samples, look at cases with high score variance. Common fixes:
- Split a vague criterion (“quality”) into two measurable ones.
- Cap the reasoning length so the model does not ramble into unrelated concerns.
- Pin the model version. Provider swaps change judge behavior even at temperature 0.
Adjust the USER_TEMPLATE to add explicit negative examples from your outlier set. This is iterative, not a one-shot config.
Step 7: Verify success
You verify a judge the same way you verify any measurement instrument: against a ground truth and against itself.
First, label 30–50 items by hand or with a senior reviewer. Run the judge on those items. Acceptable performance: within ±1 point of human mean on at least 90% of items, and judge-to-judge std dev under 0.5 on repeated runs.
Second, run the consistency check from Step 5 on a rotating sample daily. If std creeps above threshold, your provider changed something or your prompt drifted.
python -m pytest tests/test_judge_consistency.py --threshold=0.5A passing suite means your chain-of-thought prompting LLM judge is stable enough to gate production prompts.
Caveats engineers miss
Chain-of-thought is not free. You pay per output token for the reasoning text, and latency triples versus a bare score. Strip the reasoning before sending judge results to a dashboard, but archive it for audit. Also, CoT can entrench a wrong premise if the rubric is biased—reasoning makes the bias more readable, not less real.
Use the judge to compare candidates, not to produce absolute truth. Pair it with spot-checking and you get a eval loop that survives contact with real traffic.