n4nAI
Open console →
Developer toolsTutorialLLM-as-a-Judge Techniques

Using GPT-4o as a judge for summarization quality

Step-by-step tutorial: implement GPT-4o as a judge for summarization quality with Python, scoring rubrics, and fallback for rate limits via API.

n4n Team3 min read627 words

Audio narration

Coming soon — every post will get a voice note here.

Evaluating summarization output is messy because traditional metrics like ROUGE correlate weakly with human judgment. Using GPT-4o as a judge for summarization gives you a scalable, reasonably aligned signal if you constrain the prompt and enforce a strict scoring schema.

Prerequisites

  • Python 3.10 or newer
  • openai Python package (>=1.30)
  • An OpenAI API key (or an OpenAI-compatible endpoint)
  • A small corpus of source documents and candidate summaries to test

Set the key in your environment:

export OPENAI_API_KEY="sk-..."

Install the client:

pip install openai==1.40.0

Why not just use ROUGE or BERTScore

ROUGE measures n-gram overlap. A summary that paraphrases accurately but uses different words scores poorly. BERTScore catches semantic similarity but still rewards verbose, redundant text. Human reviewers care about faithfulness and coverage, not lexical match. Using GPT-4o as a judge for summarization lets you encode those human criteria directly into the prompt and get a numeric trace.

Define the evaluation rubric

A judge model drifts if you ask for a single “quality” score. Break quality into three independent dimensions and force integer scores:

  • faithfulness (1-5): Does the summary invent facts not in the source?
  • coverage (1-5): Does it capture the key points?
  • conciseness (1-5): Is it free of redundant filler?

We encode this as a JSON schema so GPT-4o returns parseable output. Structured outputs are non-negotiable when you run this in a pipeline.

{
  "type": "object",
  "properties": {
    "faithfulness": { "type": "integer", "minimum": 1, "maximum": 5 },
    "coverage": { "type": "integer", "minimum": 1, "maximum": 5 },
    "conciseness": { "type": "integer", "minimum": 1, "maximum": 5 },
    "rationale": { "type": "string" }
  },
  "required": ["faithfulness", "coverage", "conciseness", "rationale"]
}

Build the judge function

We use the structured outputs feature to guarantee the schema. The system prompt sets the role; the user prompt injects source and summary. Temperature zero reduces variance between repeated calls.

from openai import OpenAI
import os, json

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

JUDGE_SCHEMA = {
    "type": "object",
    "properties": {
        "faithfulness": {"type": "integer", "minimum": 1, "maximum": 5},
        "coverage": {"type": "integer", "minimum": 1, "maximum": 5},
        "conciseness": {"type": "integer", "minimum": 1, "maximum": 5},
        "rationale": {"type": "string"}
    },
    "required": ["faithfulness", "coverage", "conciseness", "rationale"]
}

def judge_summary(source: str, summary: str) -> dict:
    resp = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "You are a strict evaluation judge. Score the summary against the source on three dimensions using the provided schema. Never give partial scores."},
            {"role": "user", "content": f"SOURCE:\n{source}\n\nSUMMARY:\n{summary}"}
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {"name": "judge_score", "schema": JUDGE_SCHEMA}
        },
        temperature=0
    )
    return json.loads(resp.choices[0].message.content)

Prepare test data

Use a concrete example so you can see the judge behave.

source_doc = """
The quarterly earnings call revealed revenue of $3.2B, up 14% YoY.
The cloud segment grew 22%, offsetting a 3% decline in hardware.
Management guided next quarter slightly lower due to supply constraints.
"""

good_summary = "Revenue rose 14% YoY to $3.2B, driven by 22% cloud growth. Hardware dipped 3%. Next-quarter guidance is marginally lower on supply issues."

bad_summary = "The company had a great quarter. Everything is up and the future looks bright."

Run the judge

good_score = judge_summary(source_doc, good_summary)
bad_score = judge_summary(source_doc, bad_summary)

print("GOOD:", good_score)
print("BAD:", bad_score)

Expected output resembles:

GOOD: {"faithfulness": 5, "coverage": 5, "conciseness": 4, "rationale": "Summary accurately reflects all key metrics and guidance; slightly terse but acceptable."}
BAD: {"faithfulness": 2, "coverage": 2, "conciseness": 3, "rationale": "Lacks specific figures and introduces vague positivity not grounded in source."}

The judge should separate the two clearly. If it doesn’t, tighten the system prompt or add few-shot examples.

Few-shot calibration

If zero-shot scoring looks noisy, prepend two worked examples to the user message. This anchors the scale.

FEW_SHOT = [
    {"role": "user", "content": "SOURCE: Cats are mammals. SUMMARY: Cats are reptiles."},
    {"role": "assistant", "content": '{"faithfulness":1,"coverage":1,"conciseness":3,"rationale":"False claim about reptiles"}'},
    {"role": "user", "content": "SOURCE: Cats are mammals. SUMMARY: Cats are mammalian pets."},
    {"role": "assistant", "content": '{"faithfulness":5,"coverage":3,"conciseness":5,"rationale":"Correct and concise"}'}
]

def judge_summary_fewshot(source: str, summary: str) -> dict:
    messages = [
        {"role": "system", "content": "You are a strict evaluation judge. Score via schema."}
    ] + FEW_SHOT + [
        {"role": "user", "content": f"SOURCE:\n{source}\n\nSUMMARY:\n{summary}"}
    ]
    resp = client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        response_format={"type": "json_schema", "json_schema": {"name": "judge_score", "schema": JUDGE_SCHEMA}},
        temperature=0
    )
    return json.loads(resp.choices[0].message.content)

Aggregate and compare

For a batch, collect scores and compute means per dimension.

import statistics

def batch_judge(samples):
    rows = []
    for src, summ in samples:
        s = judge_summary(src, summ)
        rows.append(s)
    agg = {
        dim: statistics.mean(r[dim] for r in rows)
        for dim in ["faithfulness", "coverage", "conciseness"]
    }
    return agg, rows

samples = [(source_doc, good_summary), (source_doc, bad_summary)]
agg, _ = batch_judge(samples)
print(agg)

This prints average scores across the pair. Use it to rank candidate summarizer models.

Scale with fallback and metering

When you run thousands of judgments, OpenAI rate limits will bite. An OpenAI-compatible gateway such as n4n.ai provides automatic fallback when a provider is degraded and per-token usage metering, which keeps batch jobs from silently failing. Swap the client base URL:

client = OpenAI(
    api_key=os.environ["N4N_API_KEY"],
    base_url="https://api.n4n.ai/v1"
)

The same gpt-4o model string works if the gateway addresses 240+ models under one endpoint. The gateway forwards provider cache-control hints, so prefix caching on long source documents cuts cost.

Control for judge bias

LLM judges are not infallible. Three concrete failure modes to mitigate:

Position bias

If you compare two summaries, the judge favors the first. Randomize order and average.

Verbosity bias

GPT-4o sometimes rewards longer summaries. Cap summary length in your summarizer before judging.

Self-enhancement bias

A summary generated by GPT-4o may score higher from GPT-4o. Include at least one non-OpenAI model in your candidate set as a control.

Validate against human labels

Before trusting the pipeline, label 50 pairs yourself. Compute Spearman correlation between your judge scores and human rankings.

from scipy.stats import spearmanr

human_ranks = [1, 2]  # example: good ranked higher
judge_ranks = [good_score["coverage"], bad_score["coverage"]]
rho, _ = spearmanr(human_ranks, judge_ranks)
print(f"correlation: {rho:.2f}")

Aim for ρ > 0.6 on your domain. If lower, revise the rubric or switch to few-shot judging with example scoring.

Production checklist

  • Pin temperature=0 and a fixed seed if your client supports it.
  • Log the full judge rationale; debug failures by reading it.
  • Store raw JSON responses for audit.
  • Set a max tokens on the judge to avoid runaway rationale text.
  • Alert on schema validation errors; treat them as eval failures.
  • Run a weekly correlation check against a small human-labeled set.

Using GPT-4o as a judge for summarization is not a replacement for human review, but it compresses the feedback loop from days to seconds. Ship the rubric, measure correlation, and iterate.

Tagsllm-as-a-judgegpt-4osummarizationtutorial

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Developer toolsFine-tuning a smaller model to act as an LLM judgeStep-by-step tutorial on fine-tuning a smaller model as LLM judge: generate preference data, train a LoRA adapter on Mistral-7B, and validate accuracy.Developer toolsBuilding a rubric-based LLM judge promptStep-by-step guide to building a rubric-based LLM judge prompt for LLM-as-a-judge, with code examples and calibration against human labels.Developer toolsCalibrating LLM judges against human evaluation scoresA practical how-to for calibrating LLM judges against human scores: build a labeled set, fit a calibration layer, and measure agreement.Developer toolsChain-of-thought prompting for more consistent LLM judgesLearn how to build a consistent LLM judge using chain-of-thought prompting. Step-by-step guide with code for scoring and variance checks.Developer toolsCommon failure modes of LLM-as-a-judge pipelinesAn analysis of the failure modes LLM-as-a-judge pipelines encounter in production, with concrete examples and tradeoffs for engineers building eval systems.Developer toolsLLM-as-a-judge vs human evaluation: cost vs accuracyA practical head-to-head of LLM-as-a-judge vs human evaluation across cost, latency, and accuracy, with a clear verdict for engineering teams shipping LLM features.

All llm-as-a-judge techniques posts →