Most agent benchmarks report accuracy on curated datasets while ignoring the spend required to get there. The metric that actually predicts whether an AI system survives contact with production is cost per successful task AI agent, not pass rate on a leaderboard. If a task costs $2.00 to complete with 90% success but a cheaper pipeline gets 80% at $0.20, the latter wins on real workloads.
Why accuracy-only benchmarks lie
Curated datasets hide retry cost
Agents in production rarely solve a task on the first try. They call tools, parse messy output, hit rate limits, and retry. A benchmark that counts a task as “solved” if any of three attempts succeeds, but reports only the success rate, conceals the multiplier on token spend. In a swarm of 1,000 tasks, a 95% success rate with an average of 4 attempts per task burns 4x the tokens of a 90% success rate with 1.2 attempts. The former looks better on paper and bankrupts you in practice.
The SWE-bench style resolve rate is a case in point: it tells you how many issues a model closed, but says nothing about whether the run used a single careful pass or ten speculative patches each spinning up a sandbox. The token and compute bill diverges by an order of magnitude across those strategies.
Latency is a direct cost
Engineers often separate “performance” from “cost,” but latency is a direct cost in synchronous user-facing agents. A task that takes 60 seconds of model time at a given per-token rate is a different animal from one that takes 6 seconds. If your benchmark ignores wall-clock and queue time, you misprice the task. For background batch agents, latency converts to compute reservation cost; for interactive agents, it converts to user churn.
Defining cost per successful task AI agent
The formula is simple:
C = total_spend_on_task_batch / number_of_tasks_meeting_success_criteriaBut the denominator and numerator need discipline.
What counts as success
Define success as the external outcome, not model output shape. For a “file a bug report” agent, success is the ticket existing in the tracker with required fields, not a 200 response from the LLM. For a coding agent, success is a passing test suite, not a plausible diff. If you measure intermediate steps, you will optimize the wrong thing and drive cost per successful task AI agent down by cheating the definition.
Counting all spend
Include every dollar that left your account to attempt the batch:
- Prompt and completion tokens for every model call, including discarded drafts.
- Tool call overhead (API fees for search, browser, sandboxes).
- Retry loops and fallback model calls.
- Failed attempts that preceded success.
- Routing overhead if you pay per request to a gateway.
Exclude one-time research cost, but include inference cost of evaluation itself if it runs in the same pipeline. A task that fails all attempts still contributed to spend; it raises the metric, which is correct.
Measuring it in practice
You need instrumentation from the first call. Below is a minimal pattern for wrapping an OpenAI-compatible client to accumulate cost. Assume PRICING is a dict you populate from your provider’s published rates (per token or per image, etc.).
from openai import OpenAI
client = OpenAI(base_url="https://api.example.com/v1", api_key="KEY")
PRICING = {
"model-a": {"prompt": 0.000003, "completion": 0.000015},
"model-b": {"prompt": 0.000001, "completion": 0.000005},
}
def tracked_call(model, messages, attempt_budget=3):
cost = 0.0
last = None
for _ in range(attempt_budget):
resp = client.chat.completions.create(model=model, messages=messages)
u = resp.usage
cost += u.prompt_tokens * PRICING[model]["prompt"]
cost += u.completion_tokens * PRICING[model]["completion"]
last = resp
if is_valid(resp):
return cost, True, resp
return cost, False, last
batch_cost = 0.0
successes = 0
for task in tasks:
c, ok, _ = tracked_call("model-a", task.messages)
batch_cost += c
if ok:
successes += 1
cps = batch_cost / max(successes, 1)This computes cost per successful task AI agent for the batch. Note we divide by successes, not attempts. Failed tasks still contributed to spend, so they raise the metric—correctly.
Logging structured runs
Emit one JSON object per task so you can aggregate later without re-running:
{
"task_id": "t-882",
"success": true,
"attempts": 2,
"model_calls": [
{"model": "model-a", "prompt_tokens": 1200, "completion_tokens": 300},
{"model": "model-b", "prompt_tokens": 1100, "completion_tokens": 250}
],
"tool_cost_usd": 0.004,
"total_cost_usd": 0.021
}Aggregating total_cost_usd and counting success gives you the metric with zero guesswork.
Using fallback to control the numerator
Automatic fallback across providers is the highest-leverage way to cut cost without sacrificing success. If your primary model is rate-limited or returns degenerate output, routing to a cheaper secondary model on failure keeps the task moving. An inference gateway that honors client routing directives and forwards provider cache-control hints lets you encode this once. n4n.ai exposes a single OpenAI-compatible endpoint spanning 240+ models with automatic fallback and per-token metering, which turns the tracked_call above into a configuration rather than custom code.
{
"route": {
"primary": "anthropic/claude-3.5-sonnet",
"fallback": ["openai/gpt-4o-mini", "meta/llama-3-70b"],
"on_status": [429, 500, "degraded"]
},
"cache": {"ttl": 3600}
}With metering, each task’s total token count is reported back; you aggregate those into batch_cost without manual pricing tables.
Tradeoffs and where the metric misleads
High-stakes tasks invert the weight
For a medical triage agent, a 1% drop in success may cost lives; cost per successful task AI agent becomes secondary to a minimum success bar. Set a floor (e.g., must exceed 99% success) then minimize cost under that constraint. The metric is still useful, but as a constrained optimization not a raw minimizer. Publishing only the cost number without the floor is irresponsible.
Small samples lie
If you run 20 tasks, one expensive failure swings cost per task by 5x. You need hundreds of tasks per configuration to get a stable estimate. Bootstrap confidence intervals; report median and p90 cost, not just mean.
import numpy as np
costs = [task_cost(i) for i in range(500)]
print(np.median(costs), np.percentile(costs, 90))A single p90 that is 4x the median tells you the agent occasionally goes off the rails; that variance is itself a cost driver in production when a task blocks a user.
Exploration cost vs steady state
During development you intentionally try expensive models to calibrate. Those costs should be tracked separately from production runs. Mixing them inflates the apparent cost per successful task AI agent and may lead you to discard a cheap stable config because the eval included prototyping spikes. Tag runs with phase: prototype or phase: prod and filter before computing the headline number.
Metric gaming via success loosening
Because the denominator is success count, there is pressure to loosen success criteria to make the number look good. Resist. A “success” that requires human cleanup later is not success; it is deferred cost. If you cannot automate verification, at least sample human review and add its hourly cost to the numerator.
A worked comparison
Consider two configurations on the same 200-task set:
- Config X: success 92%, average 3.1 attempts, uses a frontier model exclusively.
- Config Y: success 84%, average 1.4 attempts, uses a mid model with fallback to frontier only on parse failure.
Even without exact prices, if frontier costs ~5x mid per token, Config Y’s total token volume is far lower despite lower success. The cost per successful task AI agent for Y can be half of X. The 8-point accuracy drop is real, but if the task is “draft a support reply that a human edits,” Y ships more volume per dollar. That trade is a business decision, but only visible when cost per successful task is on the table.
A decisive takeaway
Adopt cost per successful task AI agent as the headline number for any agent eval. Report it alongside success rate, latency p50/p90, and sample size. Instrument token spend from day one, use fallback routing to keep the numerator down, and never publish a leaderboard that omits cost. Accuracy without cost is a demo; cost per successful task is engineering.