Why agentic workflows amplify per-token latency costs

Agentic workflow latency amplification is the silent budget killer when you bolt a reasoning loop onto an LLM. A single completion call has a predictable time-to-first-token and decode rate; chain five or ten dependent steps and the same total output tokens can take three times longer and cost disproportionately more in prefill compute.

The arithmetic of sequential inference

A baseline non-agentic call looks like this:

latency = ttft + (output_tokens / decode_tokens_per_sec)

If a model returns 300 tokens at 30 tokens/sec with a 300 ms TTFT, you wait roughly 10.3 seconds. That is the entire interaction.

An agentic loop breaks the task into steps. Each step is its own completion call, usually with a growing message history. The latency sum becomes:

total_latency = Σ (ttft_i + output_i / decode + tool_time_i)

The dangerous part is that ttft_i is not constant. It includes prefill of the prompt, and the prompt grows every step because you resend the system prompt, prior assistant messages, and tool results. Prefill is typically billed and timed per input token. So later steps carry a heavier fixed cost before the first byte streams.

Consider a retrieval agent that takes six steps. Step one prompts 500 tokens. Step six prompts 4,000 tokens because it carries all prior context. At a realistic prefill cost of 0.5 ms per token (highly variable by provider, but directionally correct), step one pays 250 ms just to prefill; step six pays 2,000 ms. The decode portion may be similar across steps, but the fixed tax compounds. This is the core of agentic workflow latency amplification: fixed costs repeated under the guise of progress.

Why context growth compounds the tax

Most agent frameworks append rather than compress. Here is a minimal loop:

messages = [{"role": "system", "content": "You are a helpful agent."}]
for _ in range(6):
    resp = client.chat.completions.create(model="model-id", messages=messages)
    msg = resp.choices[0].message
    messages.append(msg)
    if msg.tool_calls:
        for call in msg.tool_calls:
            obs = run_tool(call)
            messages.append({"role": "tool", "content": obs})

Every iteration re-serializes the full messages list. The model re-attends to tokens it has already seen. That is pure latency overhead with no new output tokens to show for it.

Worse, long contexts often trigger different kernel paths or quantization on the provider side, quietly dropping decode speed. You are not just paying more prefill; you may be decoding slower on step six than on step one. The net effect is that per-token latency rises even when the model is nominally the same.

Tool execution idle time

Agentic workflow latency amplification is not only about model time. Tool calls insert wall-clock gaps where the model produces zero tokens.

step_latency = model_think + tool_run + model_absorb_result

If a SQL query takes 1.2 seconds, that time lands in your per-token latency denominator only if you divide total wall time by output tokens. A 20-token decision after a 1.2 s tool wait yields an effective 60 ms per token—versus maybe 15 ms per token for a steady decode. The user perceives lag; the cost dashboard shows inflated time-per-token even if token prices are unchanged.

This is why naive “agents are just more API calls” framing misses the point. The calls are interdependent and padded with non-generative waits.

Measuring per-token latency in practice

Define a metric that exposes the tax:

effective_latency_per_output_token = total_wall_ms / total_completion_tokens

Capture usage from each step. A typical step response includes:

{
  "usage": {
    "prompt_tokens": 1800,
    "completion_tokens": 35,
    "total_tokens": 1835
  },
  "latency_ms": 920
}

That step alone costs 26 ms per output token. Aggregate six steps with 200 total output tokens and 6,000 ms wall time, and you are at 30 ms per token. A single 200-token call at 300 ms TTFT and 30 tok/s is ~8.8 ms per token. The agentic path is 3.4x worse on this metric despite identical output volume.

Per-token usage metering helps here. If your gateway returns prompt and completion tokens per step, you can attribute which loop iteration blew the budget. Without that instrumentation you will blame the model.

A sample trace

The table shows per-token latency climbing from 17.5 ms to 75 ms despite similar decode. The driver is prefill and tool wait, not generation. Agentic workflow latency amplification shows up as a right-skewed curve across steps, not a flat line.

Mitigations that actually work

Collapse independent steps

If two tools have no data dependency, call them in parallel. OpenAI-compatible APIs accept multiple tool calls in one message. Execute them concurrently, append results once.

# pseudo
calls = resp.choices[0].message.tool_calls
results = await asyncio.gather(*[run_tool(c) for c in calls])

This removes sequential TTFT stacking and halves the wall time when tools are slow.

Push prefix caching

Providers support cache-control on system prompts and stable context prefixes. A gateway that forwards those hints prevents re-prefill of the system block on every step.

client.chat.completions.create(
    model="model-id",
    messages=[{"role":"system","content":"...","cache_control":{"type":"ephemeral"}}],
)

Even if only the system prompt is cached, a 300-token system block saved on five later steps recovers ~750 ms of TTFT in our earlier example.

Route sub-steps to smaller models

The planner does not need the largest model. A 7B class model can often pick a tool; a frontier model can execute the final synthesis. Client-side routing directives keep quality where it matters and cut TTFT on noisy steps.

Inference gateways that honor client routing directives—say, forcing a fast small model for the planner step—contain the damage. n4n.ai exposes one OpenAI-compatible endpoint across 240+ models and will automatically fall back when a provider is rate-limited, which avoids the worst tail latency spikes that dominate agentic workflow latency amplification.

Summarize history

Replace old tool outputs with a rolling summary after step three. You trade a little recall for a smaller prompt_tokens and thus lower prefill. This is the oldest trick and still the most reliable.

if len(messages) > 8:
    messages = [messages[0], summarize(messages[1:-2]), *messages[-2:]]

Tradeoffs: when amplification is acceptable

Amplification buys capability. A single shot prompt cannot reliably execute multi-table joins, check the result, and retry. If the agent saves two human escalations per run, 3x latency is cheap.

But many production “agents” are just chained prompts that could be one structured call. Measure first. If your effective latency per output token is 4x a baseline and task success is unchanged by removing a step, you have built latency without value.

The other tradeoff is cost. Input tokens are billed every step. Repeated 4k-token contexts at six steps means you pay for 24k input tokens to get 200 output tokens. That is a 120:1 input/output ratio, versus maybe 2:1 in a single call. The financial per-token cost is amplified exactly like the temporal one.

Decisive takeaway

Treat every agent step as a latency loan against the user’s patience. Instrument per-step prompt and completion tokens, compute effective latency per output token, and set a hard step budget. Use parallel tool calls, prefix caching, and model routing to shrink the fixed tax. If you cannot justify each step with a measurable quality gain, delete it—agentic workflow latency amplification is self-inflicted far more often than it is unavoidable.