The pass@1 metric agent evaluation convention measures whether a model produces a correct answer on the first try. That single-shot framing ignores how agents actually operate: they retry, branch, and recover across many model calls. For systems where reliability matters more than raw latency, pass@1 obscures the properties you need to optimize.
What pass@1 assumes about your system
Pass@1 comes from code-generation benchmarks where a model emits one completion and a test suite checks it. The implicit contract: one inference call, one answer, deterministic grading. Agents violate every part of that contract.
An agent is a control loop. It calls a model, observes a tool result, decides a next action, and repeats until termination. The final answer depends on the sequence of decisions, not a single token distribution. Scoring only the first model output tells you nothing about whether the system converges.
The pass@1 metric agent evaluation lens is borrowed from a world where the program is the model output. In an agent, the model output is one node in a graph.
The anatomy of an agent run
Consider a customer-support agent that must refund a transaction. It needs to:
- Parse the request.
- Call
lookup_orderwith an ID. - Verify eligibility via
policy_check. - Execute
issue_refund. - Summarize.
A single mistake at step 2 (bad ID extraction) can be corrected at step 3 if the agent notices the empty result and retries. Pass@1 would mark the run failed if the first model call produced a malformed ID, even if the agent self-healed.
def run_agent(query):
state = {"query": query}
for step in range(MAX_STEPS):
action = model.decide(state)
if action.tool == "lookup_order":
res = lookup_order(action.args["id"])
if not res:
state["error"] = "order not found"
continue # agent retries with corrected id
state["order"] = res
# ... other tools
return state.get("refund_confirmation")The loop is the product. Measuring only the first model.decide call misses the recovery path.
Why single-shot scoring breaks for multi-step agents
Retry and fallback logic
Production agents embed retry. If a provider returns 429, the agent (or its gateway) retries on a different model. n4n.ai offers automatic fallback when a provider is rate-limited or degraded, which means a given logical agent run may span three providers transparently. Pass@1 evaluated against a single provider’s first response is disconnected from observed behavior.
{
"run_id": "a1",
"attempts": [
{"provider": "openai", "status": 429},
{"provider": "anthropic", "status": 200, "answer": "refund issued"}
],
"final": "success"
}If you score the first attempt, you record failure. The user got success.
Tool invocation and partial progress
Agents often make progress before failing. A data-analysis agent might correctly load a CSV, compute a pivot, then crash on a chart render. Pass@1 says zero. But the loaded dataframe and pivot are reusable state; a human would call that 80% done.
Partial credit matters when agents are composed. A planner agent that picks the right tools but passes wrong params to one of them is more valuable than one that picks random tools.
Non-determinism and temperature
At temperature > 0, the same prompt yields different first tokens. Pass@1 becomes a Monte Carlo estimate requiring many samples to stabilize. For an agent, you care about the distribution of end states, not the first token’s correctness.
Better metrics for agent evaluation
success@k and budgeted success
Borrow from retrieval: measure success within k attempts or within a token budget. Define success@k as the probability the agent reaches the goal within k model calls. More useful: success@cost — did it finish under a max token spend?
def success_at_k(trajectories, k, goal_reached):
wins = sum(1 for t in trajectories if goal_reached(t) and len(t.calls) <= k)
return wins / len(trajectories)Trajectory cost and token accounting
Agents burn tokens per step. A run that succeeds at pass@1 but costs 50k tokens is worse than one that recovers from a stumble in 10k. Per-token usage metering lets you compute cost-weighted success.
cost = sum(call.prompt_tokens + call.completion_tokens for call in trajectory)
efficient = goal_reached(trajectory) and cost < BUDGETPartial credit and state reachability
Define a state graph. Award credit for reaching intermediate states. If the goal is unreachable but the agent reached the penultimate state, assign 0.9. This aligns with how you’d debug: you fix the last hop, not the whole plan.
def score_trajectory(traj):
if traj.final.get("refund_confirmation"):
return 1.0
if traj.final.get("order_loaded") and traj.final.get("policy_ok"):
return 0.7
if traj.final.get("order_loaded"):
return 0.4
return 0.0A worked example: booking agent
Suppose an agent books a flight. The ideal trajectory:
- Parse dates
- Search inventory
- Select flight
- Pay
- Confirm
In testing, model call 1 misformats the date. The agent detects the search error, reformats, and continues. Pass@1 flags this as a fail. Trajectory scoring records: date parsed (0.2), search succeeded after retry (0.5), payment (0.8), confirm (1.0). The run succeeded; the metric should reflect that.
If you only track pass@1, you will either over-prompt (force giant system messages to get first-call perfection) or ship an agent that looks weak in eval but works fine. Neither helps.
Tradeoffs of abandoning pass@1
Trajectory metrics are heavier. You must log every step, replay tool calls, and define goal predicates. That’s real instrumentation work. Pass@1 is cheap: one call, one grade.
Also, pass@1 is comparable across papers. If you publish success@cost, readers need your budget definition. Standardization is weaker.
But the alternative is shipping blind. An agent that scores 0.9 pass@1 but fails 30% of the time after step 3 will surprise users. The eval must match the deployment surface. The pass@1 metric agent evaluation habit persists because it is easy, not because it is right.
Implementing trajectory eval
Capture the full run as a list of structured events. Grade offline.
class Trajectory:
calls: list[ModelCall]
tool_results: list[ToolResult]
final_state: dict
def evaluate(run: Trajectory):
if run.final_state.get("refund_confirmation"):
return 1.0
if run.final_state.get("order_loaded"):
return 0.5
return 0.0Run hundreds of seeded scenarios. Aggregate with variance. Report success@k and median cost. Store the trajectories so you can recompute scores when your goal definition tightens.
A minimal harness:
python eval_agent.py \
--scenarios ./golden.jsonl \
--max-steps 12 \
--budget-tokens 30000 \
--out report.jsonThe report should include not just means but p90 cost and failure-mode histogram.
Decisive takeaway
Stop using the pass@1 metric agent evaluation as a primary signal for agent systems. It measures a property your agent does not have: single-shot determinism. Score trajectories—success within budget, partial state credit, and retry resilience. The instrumentation cost pays back the first time a “high pass@1” agent fails in production because its recovery path was never measured.