n4nAI
Open console →
Developer toolsDefinitionLLM-as-a-Judge Techniques

What is LLM-as-a-judge, and when should you trust it

What is LLM-as-a-judge? This explainer defines using language models to score or judge AI outputs, covers how it works, when to trust it, and pitfalls.

n4n Team4 min read831 words

Audio narration

Coming soon — every post will get a voice note here.

What is LLM-as-a-judge? It is the practice of using a language model to score, rank, or critique the outputs of another model or system, treating the evaluator model as an automated proxy for human judgment. The approach has become a standard component in offline evaluation pipelines where human review cannot keep pace with generation volume.

How LLM-as-a-Judge Works

The mechanism is straightforward: feed a candidate output, optional context, and a rubric into a judge model, then parse a structured verdict. The devil is in the prompt design, model selection, and output parsing.

Prompt Construction

A judge prompt must specify criteria, scale, and output schema. Ambiguity produces low inter-rater reliability and makes scores non-comparable across runs.

judge_prompt = """
You are a strict evaluator. Given a user question, a reference answer, and a candidate answer, score the candidate from 1 to 5 on factual accuracy.

User question: {question}
Reference: {reference}
Candidate: {candidate}

Respond ONLY with JSON: {"score": int, "reason": str}
"""

Version this prompt like code. A single word change (“helpful” vs “correct”) shifts score distributions. Store the prompt hash alongside results.

Model Selection

The judge should match or exceed the capability of the system under test on the measured dimension. A small model judging a large model’s complex reasoning will miss errors and report false confidence.

from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")

resp = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=[{"role": "user", "content": judge_prompt.format(
        question="What is the capital of France?",
        reference="Paris",
        candidate="The capital is Paris."
    )}],
    response_format={"type": "json_object"}
)
print(resp.choices[0].message.content)

The gateway forwards the request to the named model and applies fallback if the provider is degraded. For judge loops that run thousands of times per hour, that fallback is not optional.

Output Parsing

Expect malformed JSON. Wrap calls in retry logic and validate types.

import json, re

def parse_judge(text):
    try:
        return json.loads(text)
    except json.JSONDecodeError:
        m = re.search(r'\{.*\}', text, re.DOTALL)
        return json.loads(m.group(0)) if m else {"score": None}

Variants of the Technique

Understanding what is LLM-as-a-judge requires distinguishing the common topologies.

Absolute Scoring

A single output is scored against a rubric. Cheap and parallelizable, but sensitive to the judge’s interpretation of the scale. Use it for format compliance (“is valid JSON?”) more than subtle quality.

Pairwise Comparison

Two candidates are presented and the judge picks a winner. This reduces verbosity bias and yields more stable signals because the model only needs relative judgment.

{
  "candidate_a": "Paris is the capital.",
  "candidate_b": "France's capital city is Paris.",
  "instruction": "Which is more concise and correct? Reply {\"winner\": \"A\"|\"B\"}"
}

Reference-Free Evaluation

No gold answer is supplied; the judge uses internal knowledge. Useful for open generation, but riskier because errors in the judge’s own knowledge propagate.

Why It Matters

Human labeling is slow and expensive. LLM-as-a-judge provides a reproducible signal for:

  • Regression testing in CI: fail a prompt change that drops average score by more than a threshold.
  • Pairwise A/B tests at scale without exposing both variants to users.
  • Continuous monitoring of production traffic samples for drift.

It does not require annotators to read every output, only to validate the judge periodically. Teams that ship weekly prompt changes use judges to block bad deploys the same way unit tests block broken code.

A Concrete Example

Consider a RAG pipeline over internal docs. You suspect the retriever sometimes returns stale policy text. You sample 200 questions, generate answers with the current and previous index, and judge pairwise.

eval_pairs = [
  {"question": "VPN setup steps?", "old": "Use Cisco AnyConnect.", "new": "Use FortiClient v7."},
  # ... 199 more
]

winners = []
for pair in eval_pairs:
    judge_msg = (
        f"Question: {pair['question']}\n"
        f"Old: {pair['old']}\nNew: {pair['new']}\n"
        f"Which is more accurate per current policy? JSON {{\"winner\":\"old\"|\"new\"}}"
    )
    r = client.chat.completions.create(
        model="anthropic/claude-3.5-sonnet",
        messages=[{"role":"user","content":judge_msg}],
        response_format={"type":"json_object"}
    )
    winners.append(parse_judge(r.choices[0].message.content)["winner"])

new_wins = sum(1 for w in winners if w == "new")
print(f"New wins: {new_wins}/200")

If new wins 85% with a binomial test p<0.01, ship the index update. This catches regressions human reviewers would miss in manual QA queues.

Common Misconceptions

It Replaces Human Evaluation

It replaces the first-pass filter. You still need human spot-checks, especially for nuanced dimensions like tone appropriateness or cultural sensitivity.

Higher Score Means Better

Judge models exhibit position bias (preferring the first item), verbosity bias (longer answers score higher), and self-preference if the judge generated the candidate. Randomize order and use a separate judge model from the generator.

Any Model Works

A 7B model cannot reliably judge a 70B model’s quantum mechanics explanation. Capability gap invalidates scores. Match the judge to the task difficulty.

What is LLM-as-a-judge useful for in absolute terms?

Mostly relative comparison. Absolute scores are not calibrated to human utility without a mapping step. Treat a score of 4 as “better than 3” not “80% good”.

Judges Are Objective

They encode the rubric author’s priors. If the prompt says “concise”, verbosity is penalized; if it says “thorough”, the same answer loses. The judge is a mirror of your instructions.

When Should You Trust It

Trust the judge when:

  • The task has a clear rubric (correctness, format compliance, safety violations).
  • You have validated agreement on a human-labeled holdout (Cohen’s kappa > 0.6 or Pearson > 0.8).
  • You use it for ranking, not as ground truth.

Do not trust it for:

  • Legal or medical sign-off without expert review.
  • Measuring creativity or emotional support quality.
  • Low-resource languages where the judge model is weak.

Combatting Bias

Practical steps:

  1. Randomize A/B order per sample.
  2. Use multiple judges and majority vote on borderline cases.
  3. Strip identifying metadata (timestamps, user IDs) from candidates.
  4. Calibrate score thresholds against human labels monthly.

Cost and Latency

Judging every production output doubles inference cost. Common pattern: sample 1–5% of traffic, judge asynchronously, alert on anomalies. Use a smaller judge for coarse filtering, a larger judge for borderline cases.

Integration Checklist

  • Pin judge model version and prompt hash.
  • Log full request/response for audit.
  • Maintain a human-labeled calibration set.
  • Prefer pairwise over absolute when possible.
  • Route through a gateway with fallback to avoid rate limits.
  • Set score-delta thresholds in CI to block regressions.
Tagsllm-as-a-judgedefinitionevaluationmethodology

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Developer toolsReducing position bias in LLM-as-a-judge comparisonsStep-by-step methodology for reducing position bias LLM judge comparisons including prompt swapping, multi-judge ensembles, and Python code.Developer toolsBuilding a rubric-based LLM judge promptStep-by-step guide to building a rubric-based LLM judge prompt for LLM-as-a-judge, with code examples and calibration against human labels.Developer toolsCalibrating LLM judges against human evaluation scoresA practical how-to for calibrating LLM judges against human scores: build a labeled set, fit a calibration layer, and measure agreement.Developer toolsCommon failure modes of LLM-as-a-judge pipelinesAn analysis of the failure modes LLM-as-a-judge pipelines encounter in production, with concrete examples and tradeoffs for engineers building eval systems.Developer toolsMulti-judge ensembles for more reliable LLM evaluationStep-by-step guide to multi-judge ensembles LLM evaluation: assemble diverse judges, aggregate scores, measure agreement, and avoid deployment pitfalls.Developer toolsSelf-preference bias in LLM-as-a-judge scoringSelf-preference bias in LLM-as-a-judge scoring distorts eval metrics. Learn how to detect it and architect unbiased judge pools with concrete code.

All llm-as-a-judge techniques posts →