n4nAI
Open console →
Developer toolsGuideLLM-as-a-Judge Techniques

Multi-judge ensembles for more reliable LLM evaluation

Step-by-step guide to multi-judge ensembles LLM evaluation: assemble diverse judges, aggregate scores, measure agreement, and avoid deployment pitfalls.

n4n Team3 min read684 words

Audio narration

Coming soon — every post will get a voice note here.

Single LLM-as-a-judge scores are noisy and easily biased toward their own training distribution. Multi-judge ensembles LLM evaluation fixes this by combining independent perspectives, but only if you assemble and aggregate them deliberately. This guide gives an ordered, shippable path from rubric to production.

1. Define what you are actually measuring

Before calling any model, write a spec. For a summarization task, dimensions might be factuality, coverage, and conciseness. Each gets a discrete scale and written anchors.

{
  "criterion": "factuality",
  "scale": [1, 2, 3, 4, 5],
  "anchors": {
    "1": "Hallucinates entities or events",
    "3": "Minor inaccuracies, no major distortion",
    "5": "Every claim traceable to source"
  }
}

Store the spec in version control. Rubric drift is the silent killer of evaluation pipelines.

Separate dimensions, don’t mash them

A single “overall quality” score invites judges to weight tone and correctness inconsistently. Run one judge call per dimension, then combine. This separation is the foundation of reliable multi-judge ensembles LLM evaluation.

2. Select judges that disagree productively

Diversity beats raw capability. A roster should span training corpora and parameter scales. Example set:

  • A frontier API model (e.g., gpt-4o)
  • A 70B-class open-weight model (e.g., mixtral-8x22b)
  • A 7–13B critique-tuned model (e.g., llama-3.1-8b-instruct)

If all three share a base architecture, their errors correlate. Use a unified client to avoid SDK sprawl. n4n.ai provides one OpenAI-compatible endpoint covering 240+ models with automatic fallback when a provider is rate-limited, so you can rotate judges without rewriting HTTP code.

from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="KEY")

JUDGES = ["gpt-4o", "mixtral-8x22b", "llama-3.1-8b-instruct"]

def call_judge(model, system, user):
    r = client.chat.completions.create(
        model=model,
        messages=[{"role":"system","content":system},
                  {"role":"user","content":user}],
        response_format={"type":"json_object"}
    )
    return json.loads(r.choices[0].message.content)

Avoid same-family traps

Check model cards. A “mini” version of a frontier model is not independent. Prefer different tokenizers and post-training recipes. If you cannot confirm independence, treat correlated judges as a single vote.

3. Design the judge prompt and schema

Keep the system prompt identical across judges. Vary only the model field. Demand strict JSON.

SYSTEM = """You are a strict evaluator. Score the response on the criterion using only the provided scale.
Output JSON: {"score": int, "rationale": str}. No other text."""

Force structured output

Use response_format or grammar constraints. Parsing free text wastes engineering cycles and introduces parse errors that masquerade as low scores.

Counter leniency bias

Judges default to middle scores. Add an instruction: “Scores of 3 should be rare; use the full range.” Log score distributions; if mean sits at 3.2 ± 0.3, tighten anchors.

4. Aggregate scores with explicit weighting

Start with unweighted median for robustness. Then improve with calibration on a gold set.

import numpy as np

def median_score(scores: dict) -> float:
    return float(np.median(list(scores.values())))

# Build gold set: 50 samples scored by humans
def learn_weights(gold):
    w = {}
    for m, pairs in gold.items():
        err = np.mean([abs(j-t) for j,t in pairs])
        w[m] = 1.0/(err + 1e-3)
    return w

def weighted_score(scores, w):
    tot = sum(w[m] for m in scores)
    return sum(scores[m]*w[m] for m in scores)/tot

Bootstrap confidence

Ensemble variance is a free uncertainty signal. Resample judge scores 1,000 times and report the 5–95 percentile spread.

def bootstrap_ci(scores, n=1000):
    arr = np.array(list(scores.values()))
    samples = [np.random.choice(arr, len(arr)).mean() for _ in range(n)]
    return np.percentile(samples, [5, 95])

5. Measure agreement before trusting the ensemble

Run 100 samples. Compute Cohen’s kappa for categorical bins, intraclass correlation (ICC) for continuous.

from sklearn.metrics import cohen_kappa_score
kappa = cohen_kappa_score(a_scores, b_scores)

Interpretation: <0.4 poor, 0.4–0.6 moderate, >0.6 acceptable. Low kappa means rubric ambiguity, not bad models.

Visualize disagreement

Plot scatter of judge A vs judge B. A tight line means redundancy; a cloud means at least one is misreading the task. Use this to drop redundant judges and cut cost.

6. Route disagreements to a tie-breaker

When max-min spread > 2 on a 5-point scale, escalate.

def needs_review(scores, threshold=2):
    vals = list(scores.values())
    return max(vals) - min(vals) > threshold

Options:

  • Senior arbiter: a larger model called only on flagged cases.
  • Human queue: attach rationales for context.

Tradeoff: human review is accurate but slows the loop. Use it sparsely on the long tail.

7. Productionize with metering and fallback

At scale, judge calls burn tokens. Use per-token usage metering to attribute cost per criterion and model. Honor client routing directives to pin a judge to a region, and forward provider cache-control hints to avoid recomputing identical prompts.

resp = client.chat.completions.create(
    model="mixtral-8x22b",
    messages=msgs,
    extra_headers={"x-routing": "provider:abc", "cache-control": "max-age=3600"}
)
print(resp.usage.total_tokens)

Automatic fallback ensures a degraded provider doesn’t stall your eval job. This operational layer is where a unified gateway earns its keep.

Common pitfalls and tradeoffs

Correlated judges. Three models from the same family give false confidence. Compute pairwise Pearson; drop any pair > 0.9.

Rubric drift. Prompts evolve; anchors silently shift. Version the rubric and store hash with each run.

Latency. Sequential judge calls triple p99. Wrap in asyncio or thread pool.

import concurrent.futures
def parallel_judge(models, system, user):
    with concurrent.futures.ThreadPoolExecutor() as ex:
        return list(ex.map(lambda m: call_judge(m, system, user), models))

Cost. Small models are 10–50x cheaper. Route 80% of traffic to them; reserve frontier models for tie-breaks.

Over-reliance. Ensembles reduce but don’t erase bias. Run a monthly human audit of 200 samples to catch systemic misses.

Score scaling. Don’t average across dimensions without normalization. A 1–5 factuality and 1–3 tone need mapping first.

Multi-judge ensembles LLM evaluation is a discipline, not a drop-in library. Ship the pipeline, measure agreement, and iterate on the rubric. The reliability gain is worth the operational overhead.

Tagsllm-as-a-judgeensemblesreliabilityevaluation

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Developer toolsBuilding a rubric-based LLM judge promptStep-by-step guide to building a rubric-based LLM judge prompt for LLM-as-a-judge, with code examples and calibration against human labels.Developer toolsReducing position bias in LLM-as-a-judge comparisonsStep-by-step methodology for reducing position bias LLM judge comparisons including prompt swapping, multi-judge ensembles, and Python code.Developer toolsWhat is LLM-as-a-judge, and when should you trust itWhat is LLM-as-a-judge? This explainer defines using language models to score or judge AI outputs, covers how it works, when to trust it, and pitfalls.Developer toolsWhen to use LLM-as-a-judge instead of exact-match scoringA practical guide to deciding between LLM-as-a-judge vs exact-match scoring for evaluation pipelines, with code, tradeoffs, and pitfalls.Developer toolsCalibrating LLM judges against human evaluation scoresA practical how-to for calibrating LLM judges against human scores: build a labeled set, fit a calibration layer, and measure agreement.Developer toolsChain-of-thought prompting for more consistent LLM judgesLearn how to build a consistent LLM judge using chain-of-thought prompting. Step-by-step guide with code for scoring and variance checks.

All llm-as-a-judge techniques posts →