Measuring cost per successful task, not cost per call

The instinct to track LLM spend as cost per API call is misleading for agentic systems. The unit of business value is a completed task, not a single model invocation, so teams should optimize cost per successful agent task instead. This reframing changes how you evaluate models, retries, and fallback strategies.

Why cost per call hides the real bill

A call is a single request to a model. An agent task is a goal: “refund the customer”, “summarize the contract”, “generate the migration script”. Agents rarely solve tasks in one call. They plan, call tools, reflect, and retry. If you measure only cost per call, you reward architectures that fragment work into many tiny cheap calls but fail often.

Consider a triage agent. It uses a cheap classification model to decide routing. That looks inexpensive per call. But if the classifier is wrong 25% of the time, the agent executes the wrong tool, hits a validation error, and re-runs with a stronger model. The average cost per call stays low because most calls are the cheap classifier. The cost per successful agent task includes the wasted classifier calls plus the corrective expensive call.

The shape is constant: success rate multiplies effective cost. If model A costs 1 unit per call and succeeds 70% of the time, and model B costs 3 units per call and succeeds 95% of the time, expected cost per successful agent task for A is 1/0.7 ≈ 1.43 units; for B it’s 3/0.95 ≈ 3.16 units. A is still cheaper per task. But if A succeeds only 40% and needs two retries on average, its task cost becomes (1 + 2*1)/0.4 = 7.5 units. Suddenly the “expensive” model is half the price per outcome.

Defining a successful agent task

You cannot compute cost per successful agent task without a definition of success. This is the hard part. Success may be deterministic: the database row exists, the HTTP 200 returned, the unit test passed. Or it may require an eval: a reviewer model scores the output above a threshold.

If you ship an agent without a success signal, you will optimize call cost while task success silently degrades. Instrument the task boundary first. Wrap the agent run in a function that returns a typed result:

from typing import TypedDict

class AgentResult(TypedDict):
    task_id: str
    success: bool
    detail: str

def run_refund_agent(task_id: str, user_id: str) -> AgentResult:
    # ... agent loop ...
    if db.refund_recorded(user_id):
        return {"task_id": task_id, "success": True, "detail": "refund ok"}
    return {"task_id": task_id, "success": False, "detail": "failed after retries"}

That boolean is the denominator of your metric.

Instrumenting task-level cost attribution

Once success is defined, attribute token usage to the task ID. Most OpenAI-compatible responses include a usage object. Accumulate it across every call the agent makes, including retries and tool-call completions.

from dataclasses import dataclass, field

@dataclass
class TaskAccounting:
    task_id: str
    calls: int = 0
    prompt_tokens: int = 0
    completion_tokens: int = 0
    latency_ms: int = 0
    success: bool | None = None
    per_model: dict[str, dict[str, int]] = field(default_factory=dict)

    def record(self, model: str, usage: dict, latency_s: float):
        self.calls += 1
        self.prompt_tokens += usage.get("prompt_tokens", 0)
        self.completion_tokens += usage.get("completion_tokens", 0)
        self.latency_ms += int(latency_s * 1000)
        m = self.per_model.setdefault(model, {"prompt": 0, "completion": 0})
        m["prompt"] += usage.get("prompt_tokens", 0)
        m["completion"] += usage.get("completion_tokens", 0)

    def close(self, success: bool):
        self.success = success

A gateway that emits per-token metering and honors client routing directives simplifies this. For example, n4n.ai exposes a single OpenAI-compatible endpoint with automatic fallback when a provider is degraded; because it meters tokens per request and forwards cache-control hints, you can correlate every retry under one task ID without building multi-provider reconciliation yourself.

Pricing differs per model. Compute task cost from the per-model map:

PRICING_PER_1K = {
    "gpt-4o-mini": {"prompt": 0.00015, "completion": 0.0006},
    "gpt-4o": {"prompt": 0.005, "completion": 0.015},
    "claude-3-haiku": {"prompt": 0.00025, "completion": 0.00125},
}

def cost_usd(acc: TaskAccounting) -> float:
    total = 0.0
    for model, used in acc.per_model.items():
        price = PRICING_PER_1K.get(model, {"prompt": 0.0, "completion": 0.0})
        total += (used["prompt"] / 1000) * price["prompt"]
        total += (used["completion"] / 1000) * price["completion"]
    return total

Now your dashboard can show sum(cost_usd) / count(success=true) per task type.

The retry and fallback trap

Agents fail. Networks time out. Providers rate-limit. A naive client throws and you log a failed call. A resilient agent retries, possibly with a different model. Those hidden calls are invisible if you only watch call cost.

Automatic fallback is a double-edged sword. It improves task success, which lowers cost per successful agent task, but it can quietly route a cheap task to an expensive model when the primary is degraded. Without task-level attribution you cannot tell whether fallback saved the task or blew the budget.

Consider this aggregated usage for one task:

{
  "task_id": "txn_8821",
  "calls": [
    {"model": "claude-3-haiku", "prompt_tokens": 200, "completion_tokens": 50},
    {"model": "gpt-4o", "prompt_tokens": 240, "completion_tokens": 80}
  ],
  "success": true
}

The first call hit a rate limit on the cheap provider; the gateway fell back to a larger model. The per-call average looks skewed by the cheap call, but the task succeeded at a known combined cost. That is the number to optimize.

Tradeoffs of task-level measurement

Task-level accounting is not free. You must propagate a task ID through every span, including async tool calls. You must define success, which may require building eval harnesses or deterministic checks. For low-volume internal agents, the overhead may exceed the savings.

There is also a latency coupling. Cheaper models are often faster, but if they fail and trigger retries, tail latency grows. Cost per successful agent task ignores latency unless you include it as a constraint. Engineer for a Pareto frontier: acceptable latency at minimal task cost.

Finally, success definition drift is real. If you optimize too hard on a proxy metric (e.g., “tool returned 200”), the agent may learn to call the tool with dummy arguments to mark success. Keep a sample of tasks for human review.

A decisive takeaway

Track cost per successful agent task as your primary LLM spend KPI from the first prototype. Emit a task ID at the entry point, accumulate token usage and latency across all calls and retries, and divide total spend by successful tasks per type. Use a gateway that provides per-token metering and fallback transparency so cross-provider retries stay attributable. Stop celebrating low cost per call; celebrate low cost per outcome.

That is the metric that survives contact with production.