Shipping an agent that chains tool calls across many turns without a test plan is how you get silent data corruption. To evaluate multi-step agent workflows before they reach users, you need a harness that replays real traces, scores intermediate steps, and blocks regressions in CI. This guide gives you an end-to-end pipeline you can implement with off-the-shelf libraries and a single LLM gateway.
Step 1: Define per-step evaluation criteria
A final-answer accuracy check hides broken middle steps. Break the workflow into discrete stages—planning, tool selection, argument construction, observation parsing, and final synthesis—and write a measurable bar for each. Treat these as contract tests, not vibes.
{
"step": "tool_selection",
"must": ["call_weather_api", "no_hallucinated_tools"],
"latency_budget_ms": 1500
}Put this in a versioned eval_spec.py so the rest of the pipeline imports it. If you cannot name the step, the agent is too tangled to ship.
Verify: You have a list of step names and associated assertions that a junior engineer could read without context.
Step 2: Capture production-like traces
Instrument the agent to emit a structured log for every LLM request, tool call, and observation. Use a correlation ID per run and never log raw secrets. A flat JSONL file is enough to start.
import json, time, uuid
def trace(run_id, step, payload):
record = {
"run_id": run_id,
"id": uuid.uuid4().hex,
"ts": time.time(),
"step": step,
"payload": payload
}
with open("traces.jsonl", "a") as f:
f.write(json.dumps(record) + "\n")
# In agent loop:
trace(run_id, "tool_selection", {
"tools_called": ["call_weather_api"],
"args": {"zip": "94107"},
"model": "gpt-4o-mini"
})Run the agent against a small set of real tasks—support tickets, internal queries, whatever your domain is. You now have trajectories that reflect actual user intent, not synthetic prompts.
Verify: traces.jsonl contains entries with step fields matching your criteria from Step 1 and a stable run_id per session.
Step 3: Build a deterministic replay harness
Load traces and execute the agent logic with external tools mocked but LLM calls either live or cached. pytest keeps this clean. The mock must return the exact observation recorded in the trace so you isolate LLM behavior from backend flakiness.
import pytest, json
@pytest.fixture
def traces():
with open("traces.jsonl") as f:
return [json.loads(l) for l in f]
def mock_tool(name, args, expected):
assert name in expected["tools_called"]
return expected.get("observation", {"ok": True})
def test_replay_tool_selection(traces):
for t in traces:
if t["step"] != "tool_selection":
continue
# Re-run the agent's decision logic with recorded context
selected = agent_decide(t["payload"]["context"])
assert selected["tool"] == t["payload"]["tools_called"][0]Keep the mocks strict. If the agent asks for a tool the trace didn’t, fail fast.
Verify: pytest runs green on the captured set and fails when you mutate an expected tool name in the trace.
Step 4: Run evaluations against multiple models with fallback
Model behavior shifts between versions and providers. To evaluate multi-step agent workflows across the field, point your OpenAI-compatible client at a gateway that aggregates models and handles degradation. n4n.ai exposes one endpoint covering 240+ models and automatically falls back when a provider is rate-limited, so your overnight eval sweep doesn’t die on a 429.
from openai import OpenAI
client = OpenAI(
base_url="https://api.n4n.ai/v1",
api_key="YOUR_KEY",
)
def llm_call(model, messages, routing="cost-optimized"):
return client.chat.completions.create(
model=model,
messages=messages,
extra_headers={"x-routing": routing} # honors client directives
)Sweep a matrix: gpt-4o-mini, claude-3-haiku, mixtral-8x7b. Record which steps break under cheaper models. This tells you whether you can ship on a smaller model and where you need the heavyweight.
Verify: The eval matrix produces a CSV with per-model step failure rates and zero interrupted runs due to provider errors.
Step 5: Score with executable and model-graded checks
Exact-match assertions catch structural errors—wrong tool, malformed JSON. For reasoning quality, use a separate LLM as judge with a tight rubric. Never let the judge see the system prompt; give it the step input and output only.
def judge_step(question, agent_output):
rubric = "Does the output correctly use the tool result without adding facts? Answer yes/no."
resp = llm_call("gpt-4o-mini", [
{"role": "system", "content": rubric},
{"role": "user", "content": f"Q: {question}\nA: {agent_output}"}
])
return "yes" in resp.choices[0].message.content.lower()
def score_run(trace):
hard = trace["payload"]["tools_called"] == agent_actual_tools()
soft = judge_step(trace["payload"]["question"], trace["payload"]["output"])
return {"hard": hard, "soft": soft}Combine both: hard asserts for tool args, soft scores for coherence. A step can pass hard but score 0.4 on soft—that’s a flag for review.
Verify: Your scorer returns a numeric report where each step has a hard pass rate and a soft mean score between 0 and 1.
Step 6: Measure cost and latency per step
Per-token metering lets you attribute spend to a specific planning call versus a synthesis call. If your gateway forwards provider cache-control hints, prefix caching can cut replay cost dramatically—mark static system prompts with cache_control and reuse them across traces.
def log_cost(resp, step):
u = resp.usage
# provider returns prompt/completion tokens; gateway meters per-token
print(f"{step}: {u.prompt_tokens}pt {u.completion_tokens}ct")
if hasattr(u, "prompt_tokens_details"):
print(f" cached: {u.prompt_tokens_details.cached_tokens}")Set a budget gate: if median step cost exceeds your Step 1 latency budget translated to dollars, fail the build.
Verify: The report shows token counts segmented by step and total eval run cost under your threshold.
Step 7: Wire into CI and set regression gates
Add a GitHub Action that runs the replay suite on every PR touching agent code. Keep the trace set small in CI (50–100 runs) and run the full matrix nightly.
name: agent-eval
on: [pull_request]
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install pytest openai
- run: pytest eval/ --csv=report.csv
- run: python eval/gate.py report.csvgate.py exits non-zero if any step pass rate drops >2% from baseline or soft score drops >0.1.
import sys, csv
baseline = {"tool_selection": 0.98, "arg_build": 0.95}
with open("report.csv") as f:
for row in csv.DictReader(f):
if float(row["pass"]) < baseline.get(row["step"], 1.0) - 0.02:
sys.exit(1)Verify: A PR that degrades tool selection from 98% to 90% is blocked automatically with a nonzero exit.
Step 8: Verify success holistically
You have shipped nothing yet. Success means the pipeline is trustworthy, not that the agent looks clever. Confirm:
- The replay harness covers at least 50 real traces with full step annotations.
- Every step in Step 1 has a corresponding automated check in the suite.
- The model matrix ran without manual intervention and surfaced at least one model-specific failure.
- CI blocks regressions and posts the report as a PR comment.
Run the full suite locally one more time. If it is green and the cost report looks sane, you are ready to ship the agent behind a canary that emits the same trace schema for post-ship comparison.
Verify: pytest eval/ -q exits 0, report.csv shows no step below threshold, and the canary deploy uses the same run_id correlation for live monitoring.