Self-preference bias in LLM-as-a-judge scoring

Self-preference bias LLM judge is the tendency of a model to award higher scores to texts that resemble its own outputs, even when those texts are no better than alternatives from other models. If your evaluation harness uses a single LLM to grade generations, you are almost certainly overrating your own system and underrating competitors. This analysis shows where the bias originates, how to measure it in your pipeline, and how to build a judge architecture that keeps the metric honest.

Why the bias exists

The effect is not superstition. Models are trained with reinforcement learning from human feedback (RLHF) or similar preference optimization, which bakes in a prior toward responses that match their own token distribution. When a judge model sees a response phrased in its characteristic cadence—bullet lists, hedged disclaimers, specific connective tissue—it reads that as “helpful” because that is what its reward model rewarded.

It compounds with verbosity. Larger models often produce longer answers, and judges correlate length with correctness unless explicitly instructed otherwise. If the judge shares the generator’s verbosity habit, the home team gets a double boost.

A third driver is familiarity. Decoders assign higher likelihood to sequences they would have produced. Even when the judge is not computing likelihoods, its internal valuation of “fluent” skews toward self-similarity.

A typical contaminated pipeline

Most teams start with a single model doing double duty:

from openai import OpenAI
import json

client = OpenAI()

def score_pair(prompt: str, resp_a: str, resp_b: str) -> dict:
    sys = "You are a rigorous evaluator. Score each response 1-10 on correctness and clarity."
    user = f"User prompt: {prompt}\n\nResponse A:\n{resp_a}\n\nResponse B:\n{resp_b}"
    user += "\nReturn JSON: {\"a\": int, \"b\": int}"
    r = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role":"system","content":sys},{"role":"user","content":user}],
        response_format={"type":"json_object"}
    )
    return json.loads(r.choices[0].message.content)

If resp_a was produced by gpt-4o and resp_b by a different family, the judge is biased before it reads the content. The labels “A” and “B” add a second confound: positional bias makes A favored in many models. You now have self-preference stacked on order bias.

Positional bias makes it worse

Studies of pairwise prompting show the first-listed option wins 60–70% of ties in uncorrected judges. Combine that with self-preference and you get a metric that is doubly unsafe. Always randomize order and use neutral labels (response_1, response_2).

Measuring the contamination

You cannot manage what you do not measure. Run a small experiment:

1. Pick 100 prompts from your production traffic.
2. Generate answers with Model X and Model Y (different families).
3. For each pair, have Judge X and Judge Y score both, with order randomized and labels swapped.
4. Compute average score given by Judge X to X-generated vs Y-generated, and same for Judge Y.

A simple aggregation:

import statistics

def bias_gap(scores: list[tuple[str, str, float, float]]):
    # scores: (judge_family, generator_family, score_to_own, score_to_other)
    gaps = []
    for judge, gen, own, other in scores:
        if judge == gen:
            gaps.append(own - other)
    return statistics.mean(gaps)

# Example input: judge "openai" scored its own output 8.1, other's 7.4
data = [("openai","openai",8.1,7.4), ("anthropic","anthropic",7.9,7.2)]
print(bias_gap(data))  # 0.7

If that mean gap is consistently above zero, you have quantified self-preference bias LLM judge behavior in your own stack. Published studies report gaps ranging from 0.2 to 0.8 on a 10-point scale depending on task; your numbers will vary, but any positive gap is a metric you cannot trust.

Mitigation 1: Heterogeneous judge panels

The cheapest structural fix is to never use one judge. Use at least two models from disjoint training lineages. Average their scores after blind grading.

{
  "eval_config": {
    "judges": ["openai/gpt-4o-mini", "anthropic/claude-3-5-sonnet"],
    "rounds": 2,
    "randomize_order": true,
    "label_set": ["response_1", "response_2"]
  }
}

When you call the models, strip any metadata and present both responses in identical markdown templates. A gateway that honors client routing directives—such as n4n.ai—lets you pin these two judges per batch and collect per-token usage without writing separate client logic for each provider.

The panel reduces bias because the home-field advantage of one judge is cancelled by the opposite advantage of the other. The averaged score correlates better with human preference than either alone, provided the judges are both reasonably capable.

Mitigation 2: Provenance blinding

Even with separate judges, style leaks identity. A Claude answer and a Gemini answer are distinguishable to a trained reader, and LLMs are trained readers. You can reduce leakage by normalizing surface form:

• Remove leading “Sure! Here is…” and trailing summaries.
• Convert both to plain text, same wrapping width.
• Use neutral labels (response_1, response_2) and swap order per trial.

import re

def neutralize(text: str) -> str:
    text = re.sub(r"^(sure|certainly|here('s| is)).*?\n", "", text, flags=re.I)
    text = re.sub(r"\n*(in summary|to conclude).*$", "", text, flags=re.I)
    return text.strip()

def build_trial(p1: str, p2: str, flip: bool):
    a, b = (p2, p1) if flip else (p1, p2)
    return f"Response 1:\n{neutralize(a)}\n\nResponse 2:\n{neutralize(b)}"

This is imperfect. Over-stripping can remove signal. But it raises the cost for the judge to guess ownership, which is exactly what you want.

Mitigation 3: Human calibration anchors

LLM judges are a proxy. You still need a human-labeled holdout set—50 to 200 examples scored by three independent annotators. Use it to compute a bias correction factor:

def corrected_score(raw: float, is_own: bool, k: float) -> float:
    return raw - k * (1 if is_own else 0)

where k is the average gap from your measurement step. Apply this only when using single-family judges; with a heterogeneous panel you instead compute a weighted ensemble where weights come from each judge’s correlation with the human anchor.

The tradeoff is obvious: humans cost money and time. But without anchors you have no idea whether your “unbiased” panel is actually tracking quality or just averaging two prejudices.

Tradeoffs you should accept

Running two or three judges multiplies inference cost and latency. On a 10k-example eval set, that is not trivial. You can mitigate by judging only a stratified sample, or by using a smaller judge model for first-pass filtering and a larger one only for borderline cases.

A weaker judge (e.g., a 7B model) may have lower correlation with humans but also less pronounced self-preference, because it has less distinctive style. If your budget is tight, a heterogeneous panel of mid-size models often beats a single frontier model for eval integrity.

Style normalization can occasionally hurt: if one model’s correctness is conveyed through structure that you strip, you penalize it. Keep neutralization conservative.

When self-preference is acceptable

There is one scenario where using the generator as judge is fine: pure regression testing. If you only care whether a new prompt changes output relative to last week’s same-model output, the bias is constant and cancels. But the moment you compare across models or report absolute quality, the bias is fatal.

Decisive takeaway

Self-preference bias LLM judge is not a theoretical curiosity; it is a default property of the technique. Build your evaluation system with three non-negotiables:

1. Never let the generator judge its own output. Use at least two judge models from different families.
2. Blind and randomize. Strip style tells, swap order, use neutral labels, and average over flips.
3. Anchor to humans. Maintain a small labeled set to compute bias coefficients and validate panel correlation.

Ship the panel, log the per-judge scores, and review the bias gap quarterly. If the gap widens after a model upgrade, your eval just lied to you—catch it before your product metrics do.