Writing custom evaluators with OpenAI Evals

Generic benchmarks don’t capture whether a model follows your API schema or avoids leaking PII. Writing custom evaluators OpenAI Evals gives you a structured way to encode those rules and run them against any completion model with the same CLI and reporting harness.

Prerequisites

Install the framework from source. You need Python 3.9 or newer.

git clone https://github.com/openai/evals.git
cd evals
pip install -e .

Set credentials. The CLI reads OPENAI_API_KEY by default, but you can target any OpenAI-compatible endpoint by setting OPENAI_API_BASE.

export OPENAI_API_KEY=sk-...
# or for a compatible gateway:
export OPENAI_API_BASE=https://api.example.com/v1

You also need a dataset in JSONL format where each line is a JSON object with an input and an ideal field (or whatever schema your evaluator expects).

Step 1: Define a dataset

Create data/support_bot.jsonl:

{"input": "What is your refund policy?", "ideal": "refund"}
{"input": "How do I cancel my plan?", "ideal": "cancel"}
{"input": "Talk to a human", "ideal": "human"}

This dataset tests whether a support bot mentions the expected keyword in its reply.

Step 2: Write a basic custom evaluator

Create evals/keyword_eval.py. Subclass evals.Eval and implement eval_sample. The method receives one dataset sample and a random number generator. Return a dict containing at least passed (bool).

import evals

class KeywordEval(evals.Eval):
    def __init__(self, completion_fns, **kwargs):
        super().__init__(completion_fns, **kwargs)
        self.completion_fn = completion_fns[0]

    def eval_sample(self, sample, rng):
        prompt = sample["input"]
        result = self.completion_fn(
            prompt,
            max_tokens=32,
            temperature=0,
        )
        completion = result["completion"]
        expected = sample["ideal"]
        passed = expected.lower() in completion.lower()
        return {
            "passed": passed,
            "completion": completion,
        }

The completion_fn callable follows the OpenAI Evals contract: pass a prompt string, get back a dict with a completion key. The base class handles iteration and recording.

Step 3: Register the evaluator

Open evals/registry/evals.py and add a registration call at the bottom:

from evals.registry import register

register(
    name="keyword_eval",
    eval_spec="keyword_eval.py::KeywordEval",
)

If you want to bind a default dataset, pass dataset="data/support_bot.jsonl" as a keyword argument. For this tutorial we will override the dataset on the command line.

Step 4: Run the eval

Execute the CLI. Pass the model name and your eval name. Use --dataset_jsonl to point at the file from Step 1.

oaieval gpt-3.5-turbo keyword_eval \
  --dataset_jsonl data/support_bot.jsonl

Expected output includes per-sample events and a final summary:

eval: keyword_eval
model: gpt-3.5-turbo
[ sample 1 ] passed: True
[ sample 2 ] passed: True
[ sample 3 ] passed: False
final: accuracy=0.667, n=3

If you see accuracy=1.0, your model happened to mention all keywords. Tune the dataset to be stricter.

Step 5: Build a model-graded evaluator

Keyword matching is brittle. A more robust pattern is to use a second model as a judge. Writing custom evaluators OpenAI Evals supports multiple completion functions: the first generates, the second grades.

Update evals/keyword_eval.py with a new class:

import evals

class ModelGradedEval(evals.Eval):
    def __init__(self, completion_fns, **kwargs):
        super().__init__(completion_fns, **kwargs)
        self.generator = completion_fns[0]
        self.grader = completion_fns[1]

    def eval_sample(self, sample, rng):
        gen = self.generator(sample["input"], max_tokens=64, temperature=0)
        answer = gen["completion"]

        grade_prompt = (
            f"Does the answer satisfy the intent '{sample['ideal']}'?\n"
            f"Answer: {answer}\n"
            f"Respond only with YES or NO."
        )
        grade = self.grader(grade_prompt, max_tokens=2, temperature=0)
        verdict = grade["completion"].strip().upper().startswith("Y")
        return {"passed": verdict, "completion": answer}

Register it with two models:

register(
    name="model_graded_eval",
    eval_spec="keyword_eval.py::ModelGradedEval",
    completion_fns=["gpt-3.5-turbo", "gpt-4"],
)

Run it:

oaieval gpt-3.5-turbo,model_graded_eval \
  --dataset_jsonl data/support_bot.jsonl

Note: the CLI accepts a comma-separated list of models corresponding to the completion_fns order. The first is the generator, the second is the grader.

Expected output:

eval: model_graded_eval
model: gpt-3.5-turbo,gpt-4
[ sample 1 ] passed: True
[ sample 2 ] passed: True
[ sample 3 ] passed: True
final: accuracy=1.000, n=3

Using an OpenAI-compatible gateway

If you point OPENAI_API_BASE at n4n.ai, you get one OpenAI-compatible endpoint addressing 240+ models with automatic fallback when a provider is rate-limited, which keeps long eval sweeps from dying mid-run. The eval code above does not change; only the environment variable differs.

export OPENAI_API_BASE=https://api.n4n.ai/v1
oaieval gpt-4o model_graded_eval --dataset_jsonl data/support_bot.jsonl

Step 6: Capture structured metrics

The returned dict from eval_sample is recorded as an event. You can add extra keys (e.g., latency, tokens) and they will appear in the JSONL run log under data/results. To compute aggregated metrics, rely on the passed key; the harness automatically reports accuracy.

For finer control, override run instead of eval_sample:

def run(self, recorder):
    samples = self.dataset  # loaded from registry
    for sample in samples:
        with recorder.as_test_case():
            result = self.eval_sample(sample, rng=recorder.rng)
            recorder.record(result)

This is optional; the default run already loops and records.

Debugging tips

• Print to stderr inside eval_sample; stdout is consumed by the CLI.
• Set temperature=0 for deterministic generator calls during eval development.
• Use a tiny dataset (n=3) until the logic is verified.
• If completion_fn raises, the sample is marked failed, not passed.
• The eval_spec string must be filename.py::ClassName relative to the evals package root.

Running evals in CI

Wrap the CLI call in a shell script and parse the final line. The JSONL log written to data/results contains per-sample records you can aggregate with jq:

oaieval gpt-4 model_graded_eval --dataset_jsonl data/support_bot.jsonl \
  --log_to_file data/results/run.jsonl
accuracy=$(jq -s 'map(select(.type=="final_report"))[0].accuracy' data/results/run.jsonl)

Fail the pipeline if accuracy drops below your threshold.

Wrapping up

You now have a repeatable pattern for writing custom evaluators OpenAI Evals: define a dataset, subclass evals.Eval, register the class, and run via oaieval. From keyword checks to model-graded judges, the same harness tracks results and emits accuracy. As your eval suite grows, split evaluators into separate files and keep datasets in version control so regressions are visible run-over-run.