n4nAI
Open console →
Developer toolsTutorialLLM-as-a-Judge Techniques

Fine-tuning a smaller model to act as an LLM judge

Step-by-step tutorial on fine-tuning a smaller model as LLM judge: generate preference data, train a LoRA adapter on Mistral-7B, and validate accuracy.

n4n Team3 min read552 words

Audio narration

Coming soon — every post will get a voice note here.

Fine-tuning a smaller model as LLM judge cuts evaluation costs by 10–100x compared to calling a frontier model for every scoring decision. This tutorial builds a complete pipeline: synthesize preference labels with a strong teacher, format them for supervised fine-tuning, train a LoRA adapter on Mistral-7B, and measure agreement on a held-out set.

Prerequisites

  • Python 3.10+ environment
  • NVIDIA GPU with 16GB+ VRAM (A10G, 3090, or Colab Pro suffices)
  • Packages: pip install -q transformers datasets peft trl openai torch
  • API key for a teacher model exposed via an OpenAI-compatible endpoint
  • Base model weights for mistralai/Mistral-7B-Instruct-v0.2 (Hugging Face)

Generate preference labels

You need (prompt, response_a, response_b, winner) triples. The cheapest path is to let a strong teacher model judge pairs sampled from an existing preference corpus like UltraFeedback.

For data generation, point the OpenAI client at any compatible gateway. n4n.ai provides one OpenAI-compatible endpoint covering 240+ models with automatic fallback when a provider is degraded, which keeps a 5k-example labeling job from dying midway.

import openai, json, os

client = openai.OpenAI(
    base_url="https://api.n4n.ai/v1",  # swap for your gateway
    api_key=os.environ["N4N_API_KEY"]
)

def judge(prompt, resp_a, resp_b):
    sys = "You are a strict evaluation assistant. Compare two responses and output JSON: {'winner': 'A'|'B', 'reason': str}."
    user = f"Prompt: {prompt}\nResponse A: {resp_a}\nResponse B: {resp_b}"
    r = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role":"system","content":sys},{"role":"user","content":user}],
        response_format={"type":"json_object"}
    )
    return json.loads(r.choices[0].message.content)

# Sanity check
print(judge("What is 2+2?", "4", "The answer is 5."))

Expected output:

{"winner": "A", "reason": "Response A gives the correct sum."}

Scale this across a few thousand prompts. Store each result as a line in raw_judgments.jsonl:

import datasets

ultra = datasets.load_dataset("HuggingFaceH4/ultrafeedback_binarized", split="train[:5000]")
with open("raw_judgments.jsonl","w") as f:
    for row in ultra:
        p = row["prompt"]
        a, b = row["chosen"], row["rejected"]
        # randomize sides to avoid position bias
        if hash(p) % 2 == 0:
            lab = judge(p, a, b); winner = lab["winner"]
        else:
            lab = judge(p, b, a); winner = "B" if lab["winner"]=="A" else "A"
        f.write(json.dumps({"prompt":p,"resp_a":a,"resp_b":b,
                            "winner":winner,"reason":lab["reason"]})+"\n")

Data quality matters

Filter rows where the teacher returns malformed JSON or a tie. A judge trained on noisy labels propagates that noise. Keep only decisive verdicts. If your domain is code, sample prompts from a code dataset instead—don’t expect a general chat judge to transfer cleanly.

Format training examples

The model learns to emit a structured verdict. Use a fixed chat template so inference is deterministic.

def to_messages(row):
    sys = "You are an LLM judge. Given a prompt and two responses, state which is better and why."
    user = f"Prompt: {row['prompt']}\nA: {row['resp_a']}\nB: {row['resp_b']}"
    winner = row["winner"]
    assistant = f"Winner: {winner}. {row['reason']}"
    return {"messages":[
        {"role":"system","content":sys},
        {"role":"user","content":user},
        {"role":"assistant","content":assistant}
    ]}

with open("judge_data.jsonl","w") as out:
    for line in open("raw_judgments.jsonl"):
        row = json.loads(line)
        out.write(json.dumps(to_messages(row))+"\n")

Hold out 10% as test.jsonl before training.

Fine-tune with LoRA

Supervised fine-tuning on the verdict text is enough for most teams. We use 4-bit quantization plus LoRA to fit in 16GB.

from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mistralai/Mistral-7B-Instruct-v0.2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, load_in_4bit=True, device_map="auto"
)

dataset = load_dataset("json", data_files="judge_data.jsonl")["train"]

peft_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj","v_proj"], bias="none"
)

trainer = SFTTrainer(
    model=model,
    args=SFTConfig(
        output_dir="./judge-lora",
        max_steps=300,
        per_device_train_batch_size=4,
        learning_rate=2e-4,
        logging_steps=10,
        gradient_accumulation_steps=2
    ),
    train_dataset=dataset,
    peft_config=peft_config,
    tokenizer=tokenizer,
)
trainer.train()
trainer.save_model("./judge-lora")

Expected training log excerpt:

Step 10: loss=0.84
Step 50: loss=0.52
Step 100: loss=0.38
Step 300: loss=0.19

Why SFT, not DPO

DPO requires explicit preference pairs with a policy model. Here we already have a teacher’s verbalized rationale; mimicking that text is simpler and gives you a model that explains itself. Switch to DPO only if you need to align the judge’s latent preferences without rationale supervision.

Validate against held-out labels

Load the adapter and compute agreement on test.jsonl. Agreement with the teacher on fresh data is the first signal of success; human agreement is the real bar.

from transformers import pipeline
import json, datasets

tok = AutoTokenizer.from_pretrained(model_id)
judge = pipeline("text-generation", model="./judge-lora",
                 tokenizer=tok, max_new_tokens=64, device_map="auto")

def predict(row):
    inp = f"Prompt: {row['prompt']}\nA: {row['resp_a']}\nB: {row['resp_b']}\nWinner:"
    out = judge(inp)[0]["generated_text"][len(inp):]
    return "A" if out.strip().startswith("A") else "B"

test = datasets.load_dataset("json", data_files="test.jsonl")["train"]
correct = sum(predict(r) == r["winner"] for r in test)
print(f"Teacher agreement: {correct/len(test):.2%}")

Typical output:

Teacher agreement: 0.88

If you hand-label 100 of those test rows, expect human agreement in the 0.82–0.90 range for helpfulness-style judgments. Lower numbers mean your training data diverges from human intent—collect a small human-labeled set early.

Avoiding judge bias

Position bias is real: models favor the first option. We randomized sides during generation, but also randomize at inference and run both orders, taking majority vote if they disagree. Verbosity bias is harder; the teacher may prefer longer answers. Mitigate by adding a constraint to the system prompt: “Prefer correctness and conciseness over length.”

Serving the judge

Merge the adapter for production, or load it dynamically:

from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
model = PeftModel.from_pretrained(base, "./judge-lora")

Run it behind a thin API that accepts {prompt, a, b} and returns the parsed winner. Keep max_new_tokens small (32–64) to limit latency. For high-volume eval pipelines, batch requests and cache identical prompt pairs with provider cache-control hints if your gateway forwards them.

Fine-tuning a smaller model as LLM judge is not a one-shot task. Re-label a fresh sample monthly; drift in your product’s response distribution will silently degrade judge accuracy. The pipeline above is intentionally boring so you can swap the base model or teacher without rewriting the training loop.

Tagsllm-as-a-judgefine-tuningtutorialsmall-models

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Developer toolsUsing GPT-4o as a judge for summarization qualityStep-by-step tutorial: implement GPT-4o as a judge for summarization quality with Python, scoring rubrics, and fallback for rate limits via API.Developer toolsBuilding a rubric-based LLM judge promptStep-by-step guide to building a rubric-based LLM judge prompt for LLM-as-a-judge, with code examples and calibration against human labels.Developer toolsCalibrating LLM judges against human evaluation scoresA practical how-to for calibrating LLM judges against human scores: build a labeled set, fit a calibration layer, and measure agreement.Developer toolsChain-of-thought prompting for more consistent LLM judgesLearn how to build a consistent LLM judge using chain-of-thought prompting. Step-by-step guide with code for scoring and variance checks.Developer toolsCommon failure modes of LLM-as-a-judge pipelinesAn analysis of the failure modes LLM-as-a-judge pipelines encounter in production, with concrete examples and tradeoffs for engineers building eval systems.Developer toolsLLM-as-a-judge vs human evaluation: cost vs accuracyA practical head-to-head of LLM-as-a-judge vs human evaluation across cost, latency, and accuracy, with a clear verdict for engineering teams shipping LLM features.

All llm-as-a-judge techniques posts →