How to build a custom eval suite for your AI agent

Building a custom eval suite for AI agents is non-negotiable once your agent touches production traffic. Without reproducible tasks and deterministic scoring, you are flying blind on every model swap or prompt change. This guide walks through a concrete, code-first pipeline you can stand up in a day and trust enough to block merges.

Step 1: Define eval tasks as structured cases

Start by writing a golden set of agent inputs with expected behavior. Keep each case small and isolated. A task is not a unit test for a single function; it is a scenario the agent should handle correctly end to end. Pull the first candidates from real support transcripts or user sessions, not from your imagination.

Use a flat JSONL file so you can version it in git and diff it like code:

{"id": "refund_policy_q1", "input": "What is your refund window?", "expect": {"contains": ["30 days", "receipt"], "tools": ["lookup_policy"]}}
{"id": "escalate_abuse", "input": "You are stupid and useless", "expect": {"sentiment": "neutral", "tools": ["escalate"]}}
{"id": "order_status", "input": "Where is order #123?", "expect": {"tools": ["get_order"], "contains": ["123"]}}

Load it with a few lines of Python:

import json

def load_tasks(path):
    with open(path) as f:
        return [json.loads(line) for line in f if line.strip()]

tasks = load_tasks("eval_tasks.jsonl")
print(f"loaded {len(tasks)} tasks")

The expect field is intentionally loose. You will map these to scorers in Step 3. The key is that the task file is the single source of truth for regression detection. A custom eval suite for AI agents lives or dies on the quality of this file—review it in PRs with the same scrutiny as production logic.

Do not aim for 500 tasks on day one. Ten sharp, high-frequency cases beat a broad set of vague ones. Expand only after the loop is green.

Step 2: Instrument the agent to emit traces

Your agent likely calls an LLM and maybe tools. Wrap it so eval can capture the final answer and the tool calls. Do not mock the model; run the real path. If you mock, you are testing your mock.

Below is a minimal agent loop using the OpenAI SDK. It returns a structured trace:

from openai import OpenAI

client = OpenAI()  # or point base_url at your gateway

def run_agent(user_msg, model="gpt-4o-mini"):
    messages = [{"role": "user", "content": user_msg}]
    tools = [
        {"type": "function", "function": {"name": "lookup_policy", "parameters": {}}},
        {"type": "function", "function": {"name": "escalate", "parameters": {}}},
        {"type": "function", "function": {"name": "get_order", "parameters": {}}},
    ]
    resp = client.chat.completions.create(
        model=model, messages=messages, tools=tools, tool_choice="auto"
    )
    msg = resp.choices[0].message
    trace = {
        "final_text": msg.content or "",
        "tool_calls": [tc.function.name for tc in (msg.tool_calls or [])],
    }
    return trace

If your agent uses a framework (LangChain, Pydantic AI, etc.), add a similar adapter that returns final_text and tool_calls. The eval suite should not care about internals. Capture errors too: wrap run_agent in try/except and emit a trace with error field so a crashed task is a explicit fail, not a silent skip.

For async agents, collect traces concurrently but keep task IDs mapped. Never let parallelism reorder your results—diffing requires stable identity.

Step 3: Write scoring functions

A custom eval suite for AI agents needs both hard checks and soft judges. Hard checks are fast and deterministic. Soft judges use an LLM to grade nuanced output.

Deterministic scorer:

def score_exact(trace, expect):
    if "error" in trace:
        return False
    ok = True
    if "contains" in expect:
        for phrase in expect["contains"]:
            if phrase.lower() not in trace["final_text"].lower():
                ok = False
    if "tools" in expect:
        if set(expect["tools"]) != set(trace["tool_calls"]):
            ok = False
    return ok

LLM judge for subjective criteria (e.g., tone):

def llm_judge(trace, criterion, model="gpt-4o"):
    prompt = f"Does the text meet this criterion: {criterion}?\nText: {trace['final_text']}\nAnswer YES or NO only."
    resp = client.chat.completions.create(
        model=model, messages=[{"role": "user", "content": prompt}]
    )
    return "YES" in resp.choices[0].message.content.upper()

Combine them per task:

def evaluate(task, trace):
    hard = score_exact(trace, task["expect"])
    soft = True
    if "sentiment" in task["expect"]:
        soft = llm_judge(trace, "respond with neutral professional tone")
    return {"task_id": task["id"], "hard": hard, "soft": soft, "pass": hard and soft}

Keep the judge calls cheap by using a smaller model. Cache judge responses by hashing the input text and criterion. Calibrate the judge once: run it on 20 hand-labeled traces and check agreement before trusting it in CI.

Avoid letting the judge score the same model that produced the answer without separation—use a different model family if possible to reduce self-bias.

Step 4: Run the suite against a model gateway

Execute all tasks, collect traces, and score. If you run evals across multiple providers, use one OpenAI-compatible endpoint to avoid rewriting clients. For example, n4n.ai exposes a single endpoint covering 240+ models and automatically falls back when a provider is rate-limited, so a large eval batch does not die mid-run on a 429.

from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")

def run_suite(tasks, model):
    results = []
    for task in tasks:
        try:
            trace = run_agent(task["input"], model=model)
        except Exception as e:
            trace = {"error": str(e)}
        results.append(evaluate(task, trace))
    return results

Run from bash:

python -c "
import eval_lib
tasks = eval_lib.load_tasks('eval_tasks.jsonl')
res = eval_lib.run_suite(tasks, 'gpt-4o-mini')
fails = [r for r in res if not r['pass']]
print(f'{len(res)-len(fails)}/{len(res)} passed')
exit(1 if fails else 0)
"

Set tool_choice and forward provider cache-control hints if your gateway supports them; this cuts cost on repeated judge calls. Parallelize task execution with a thread pool, but cap concurrency to respect provider limits. A custom eval suite for AI agents should finish in minutes, not hours.

Step 5: Persist results and diff over time

Write results to JSONL with a timestamp and model tag. You will compare against the previous run to catch regressions.

import json, time

def save_results(results, model, path="eval_runs.jsonl"):
    with open(path, "a") as f:
        for r in results:
            r["model"] = model
            r["ts"] = time.time()
            f.write(json.dumps(r) + "\n")

A quick diff script:

def diff_runs(current, previous):
    prev_map = {r["task_id"]: r for r in previous}
    regressions = []
    for r in current:
        if not r["pass"] and prev_map.get(r["task_id"], {}).get("pass"):
            regressions.append(r["task_id"])
    return regressions

Store the previous run in your CI cache or a small SQLite table. The point is to fail the build only on new failures, not on pre-existing known gaps. Quarantine flaky tasks: if a task fails intermittently across three runs, move it to a flaky file and fix the scorer before trusting it.

Expanding the custom eval suite for AI agents beyond happy paths means adding adversarial inputs and malformed tool outputs. Do this once the core loop is stable.

Step 6: Wire into CI

Add a GitHub Actions step that runs the suite on every PR that touches agent code or prompts:

name: eval
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.11"}
      - run: pip install openai
      - run: python run_eval.py --model gpt-4o-mini

In run_eval.py, load last green run from main branch artifact, compute regressions, and exit non-zero if any. Keep the task file in the repo so edits to evals are reviewed like code. Add a nightly scheduled job that runs the suite against the latest model snapshots to catch upstream provider drift.

How to verify success

A healthy custom eval suite for AI agents does three things: it runs in under ten minutes on a PR, it blocks merges that introduce new task failures, and it reports a clear diff of which task IDs regressed. If your suite requires manual interpretation of logs, it is not done.

Treat the eval task file as production code. When the agent’s behavior intentionally changes, update the expected field in the same PR. That keeps the suite trustworthy and prevents silent suppression of real regressions.

If you follow these steps, you will have a reproducible gate that survives model upgrades and prompt tweaks—exactly what a custom eval suite for AI agents is supposed to provide.