Golden datasets for LLM evals: how to build one

Building golden datasets for LLM evals is the highest-leverage work you can do before trusting a model in production. A golden set is a fixed, curated collection of input/output pairs or rubrics that encode the behaviors you care about, and it acts as a regression test as you swap prompts, models, or pipelines. Skip this and you are flying blind on whether your next change helped or hurt.

Start with failure modes, not coverage

Don’t try to represent all possible inputs. That path leads to a 10k-item pile that nobody trusts. Instead, write down the five to ten ways your LLM feature can embarrass you in production:

• Wrong tool call (e.g., refund triggered when policy says no)
• Leaked PII in a summary
• Refusal on a benign request
• Hallucinated citation or fact
• Broken JSON schema on extraction

Each becomes a cluster in your dataset. For a support bot, the clusters might be off_policy_refund, missing_slot, toxic_redirect. Aim for at least 15 items per cluster initially. Fewer than that and pass rates are statistical noise.

The discipline of building golden datasets for LLM evals starts with admitting you are testing for specific breaks, not generic quality.

Pull seeds from production logs

Synthetic prompts generated by another LLM lie in convenient ways. Pull real queries from your inference logs, support tickets, or staging traces. If you route traffic through a gateway, filter by route or user tier to get representative slices.

import json

seeds = []
with open("logs.jsonl") as f:
    for line in f:
        rec = json.loads(line)
        # capture errors and any human-flagged bad outputs
        if rec.get("status") == "error" or rec.get("user_flag"):
            seeds.append({
                "input": rec["prompt"],
                "meta": {"source": rec["trace_id"]}
            })

Anonymize before storing. Strip emails, tokens, customer IDs with a regex or a one-way hash. Keep the hash mapping in a separate access-controlled store if you need to pull the original later.

import re, hashlib
def scrub(text):
    text = re.sub(r"\b[\w.]+@[\w.]+\b", "[EMAIL]", text)
    text = re.sub(r"\b\d{4}-\d{4}-\d{4}\b", "[ORDER]", text)
    return text

Label with explicit rubrics

A golden dataset without a labeled “expected” or “acceptable” is just a pile of prompts. For each item, attach either a reference output or a boolean rubric. Rubrics beat references for open-ended tasks.

{
  "id": "refund-001",
  "cluster": "off_policy_refund",
  "input": "Can I get a refund for order 8821?",
  "rubric": "Must not approve refund without manager escalation. Must ask for order email.",
  "must_not_contain": ["approved", "processed refund"]
}

Bad rubric: “Should be helpful.” Good rubric: “Must extract order_id and email, output valid JSON, no extra text.” Have two labelers independently score borderline cases. Adjudicate disagreements in a short doc; that doc is your spec.

Version the file, not the folder

Store the set as a single versioned artifact: golden_v1.jsonl, golden_v2.jsonl. Use Git or a dataset registry. Never edit in place; append a new version with a changelog.

git tag -a golden-v3 -m "add 12 json-repair cases from staging"

If you use an eval harness, point it at a pinned hash. CI should fail if the golden file changes without a tag. This prevents silent drift where someone deletes the hard cases to make metrics look good.

Run against a grader, not your eyes

Manual review does not scale past 50 items. Write a deterministic checker for structural rules, and an LLM grader for semantic ones.

def check(item, output):
    for bad in item.get("must_not_contain", []):
        if bad in output.lower():
            return False, f"contained {bad}"
    return True, "ok"

For semantic match:

from openai import OpenAI
client = OpenAI()

def llm_grade(rubric, output):
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[
            {"role": "system", "content": "You are a strict grader. Answer YES or NO."},
            {"role": "user", "content": f"Rubric: {rubric}\nOutput: {output}\nSatisfies?"}
        ]
    )
    return resp.choices[0].message.content.strip().startswith("YES")

Run both. Report pass rate per cluster. Calibrate the LLM grader against 30 human-labeled items; if it disagrees with humans >10% of the time, rewrite the rubric or switch models.

Execute across model swaps

When you change the underlying model, re-run the full set. This is where an inference gateway helps: if you point your eval script at one OpenAI-compatible endpoint that fronts 240+ models with automatic fallback, you avoid writing provider-specific retry logic. n4n.ai does this and meters per-token usage so cost stays visible. You still own the grader and the dataset.

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key=KEY)
models_to_test = ["anthropic/claude-3.5-sonnet", "openai/gpt-4o"]
for m in models_to_test:
    for item in golden:
        out = client.chat.completions.create(
            model=m,
            messages=[{"role": "user", "content": item["input"]}]
        )
        # grade...

Tradeoff: LLM graders add variance. Set temperature to 0 and sample twice; flag items where grader flips. Those items are either ambiguous or expose grader fragility—fix one of the two.

Grow via triggered capture

Your set should never be “done.” Add items from:

• Production complaints and support escalations
• Eval runs where grader confidence is low or output is borderline
• New feature flags that change acceptable behavior

Automate a weekly diff:

python eval.py --model prod --baseline golden_v3.jsonl --out report.md

If a cluster drops >2% pass rate, block the deploy. Treat the golden set as a living contract.

Common pitfalls and tradeoffs

Overfitting to the grader

If you tune prompts until the LLM grader says 100%, you may have trained on the grader’s blind spots. Keep a human spot-check of 20 random items per release. If human and grader disagree, trust human and fix grader.

Golden set drift

Teams silently delete “hard” items that make numbers look bad. Protect the file in CI: any removal requires a PR with reason. Review removals like you review code.

Ignoring negative space

Include inputs that should produce refusal or empty result. A model that always answers is not safe. Add should_refuse cluster with toxic or out-of-scope prompts.

Small clusters lie

A cluster with 3 items showing 100% is noise. Minimum 15 before trusting the rate. If you can’t find 15 real examples, synthesize the rest but mark them synthetic and weight them lower in reporting.

Labeling cost vs. synthetic breadth

Human labeling is expensive but beats synthetic loops for edge cases. Use LLM-generated candidates for breadth, then human for the final 20%. For internal tools with parsed output, skip the semantic grader and use exact-match on fields.

Composition targets

A practical starting set for a mid-size feature:

• 5–8 clusters
• 15–30 items per cluster
• 30% negative/refusal cases
• 10% marked synthetic for coverage

That is roughly 150–300 items. It runs in minutes and fits in a single JSONL file.

Building golden datasets for LLM evals is iterative: define failures, seed from reality, label strictly, version, automate grading, and defend the set from drift. The dataset becomes the contract between your team and the model, and it is the only thing standing between a clever demo and a reliable system.