Building a rubric-based LLM judge prompt

A rubric-based LLM judge prompt turns subjective quality assessments into reproducible scores by giving the model explicit criteria and a fixed scale. This guide walks through building one that survives contact with production traffic, from dimension selection to calibration against human labels.

Step 1: Define evaluation dimensions and scoring scale

Pick 3–5 dimensions that map to your product’s failure modes. For a support bot, correctness, tone, and instruction-following cover most regressions. Avoid vague traits like “quality” — they correlate with nothing.

Set a fixed scale. A 1–5 integer scale with verbal anchors at each point beats a 1–10 scale because models calibrate better on fewer, distinct buckets. Never let the judge emit floats; round to integers in post-processing if it does.

Step 2: Write the rubric as machine-readable criteria

The rubric must be unambiguous. Store it as JSON so you can render it into the rubric-based LLM judge prompt and parse results consistently.

{
  "dimensions": [
    {
      "name": "correctness",
      "scale": [1, 2, 3, 4, 5],
      "anchors": {
        "1": "Contains factual errors or contradicts the user request",
        "3": "Mostly accurate with one minor omission",
        "5": "Fully accurate, complete, no hallucinations"
      }
    },
    {
      "name": "tone",
      "scale": [1, 2, 3, 4, 5],
      "anchors": {
        "1": "Hostile, dismissive, or off-brand",
        "3": "Neutral but generic",
        "5": "Warm, concise, on-brand for enterprise support"
      }
    }
  ]
}

Attach this rubric to every judge call. If you change an anchor, version the rubric file.

Step 3: Construct the judge prompt

The system message sets the role; the user message packs the rubric, the candidate output, and any reference answer. Keep the ordering fixed: rubric first, then candidate, then reference. Models attend to recent tokens, but a stable layout reduces variance.

If you point the OpenAI client at n4n.ai’s OpenAI-compatible endpoint, you get access to 240+ models and automatic fallback when a provider is rate-limited, which keeps judge runs from failing mid-evaluation.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",
    api_key="YOUR_KEY"
)

rubric_json = """
{
  "dimensions": [
    {"name": "correctness", "scale": [1,2,3,4,5],
     "anchors": {"1":"errors","3":"minor omission","5":"perfect"}}
  ]
}
"""

system_prompt = (
    "You are a strict LLM judge. Score the candidate output against the "
    "provided rubric. Output only JSON matching the schema."
)

user_content = f"""RUBRIC:
{rubric_json}

CANDIDATE:
{candidate_text}

REFERENCE:
{reference_text}

Return JSON: {{"correctness": int, "tone": int, "reasoning": str}}"""

resp = client.chat.completions.create(
    model="openai/gpt-4o-mini",
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_content}
    ],
    response_format={"type": "json_object"}
)

Step 4: Enforce structured output with a schema

JSON mode prevents free-text rambling, but it does not guarantee your keys. Validate after the call. Use a tiny parser:

import json

def parse_judge(raw):
    data = json.loads(raw.choices[0].message.content)
    assert set(data.keys()) >= {"correctness", "tone", "reasoning"}
    data["correctness"] = int(data["correctness"])
    data["tone"] = int(data["tone"])
    return data

If you need stricter contracts, use function calling with a JSON schema. The extra round-trip is worth it when the judge feeds a dashboard.

Step 5: Calibrate on a labeled sample set

A rubric-based LLM judge prompt is useless until you measure its agreement with humans. Label 50–100 real outputs yourself or pull from historical tickets with CSAT scores.

Run the judge on each sample, then compute rank correlation:

from scipy.stats import spearmanr

human = [sample["human_score"] for sample in eval_set]
judge = [parse_judge(run_judge(sample))["correctness"] for sample in eval_set]

rho, p = spearmanr(human, judge)
print(f"Spearman ρ={rho:.2f}, p={p:.3f}")

A ρ above 0.7 on correctness means the judge can flag regressions. Below that, the rubric anchors are too soft.

Step 6: Tighten ambiguous anchors

Typical failure: the judge gives 4s to everything. That signals anchor compression. Split the middle. Change “3: mostly accurate” to “3: accurate but misses one step the user explicitly asked for”. Concrete behaviors beat adjectives.

Another failure: reasoning says “good answer” but score is 1. Add a constraint in the system prompt: “The reasoning field must cite the rubric anchor that justifies the score.” This forces the model to map output to criteria.

Step 7: Automate in your pipeline

Wire the judge into a nightly eval or CI step. Load a golden set, score, and fail the build if mean correctness drops more than 0.2 from baseline.

baseline = 4.1
scores = [parse_judge(run_judge(s))["correctness"] for s in golden]
mean = sum(scores)/len(scores)
if baseline - mean > 0.2:
    raise SystemExit("Judge detected regression in correctness")

Store per-token usage if your gateway exposes it; you’ll need it to track cost as the golden set grows.

Step 8: Verify success against human labels

Verification is not “the judge runs.” Success means the rubric-based LLM judge prompt agrees with held-out human ratings and surfaces real defects.

Create a fresh set of 30 outputs the judge has never seen, labeled by a second human. Compute both Spearman ρ and raw accuracy within one point. You have a working judge when:

• ρ ≥ 0.7 on each dimension
• < 10% of items differ from human by more than 1 point
• Reasoning fields reference specific rubric anchors

If those hold, promote the rubric version and freeze the prompt. Re-calibrate monthly; model updates shift judge behavior even if your code doesn’t change.

The work is iterative. A rubric-based LLM judge prompt is a living artifact—treat it like a unit test suite, not a one-shot prompt.