A long-horizon agent latency benchmark that only records time-to-first-token misses the point: research agents spend most of their wall-clock time calling tools, waiting on rate limits, and recovering from provider errors. We argue that end-to-end task completion under realistic failure injection is the only metric that predicts production behavior, and we show how to build one.
Why token latency lies for agents
Single-shot model benchmarks report TTFT and tokens-per-second. Those numbers are irrelevant when an agent loops for 40 steps. The model inference might occupy 15% of total time; the rest is HTTP round-trips, vector search, browser automation, and exponential backoff.
Engineers optimizing the wrong metric thrash. They switch to a faster model that degrades plan quality, causing more retries and longer total latency. A proper long-horizon agent latency benchmark exposes this trap by measuring the variable that costs users: minutes until the final report lands.
Context growth compounds the lie. A research task accumulates thousands of retrieved tokens per step. Feeding a 128k context on every call inflates inference time quadratically in some implementations. Yet the user perceives latency only when the final answer stalls. Microbenchmarks never exercise that curve.
Anatomy of a long-horizon research task
Consider an agent that researches “impact of EU AI regulation on open-source models.” It must decompose the query, fetch legislation PDFs, query a vector store for prior analyses, call a search API, synthesize, critique, and write a memo.
Tool calls and waiting
Each step is a state machine transition. The agent emits a function call; the orchestrator executes it; the result returns. Over a 30-step task, even 200 ms of extra orchestration per step adds 6 seconds. That is noise compared to a search API that occasionally takes 8 seconds.
async def run_step(agent, ctx, step):
start = time.monotonic()
resp = await agent.chat(ctx.messages) # OpenAI-compatible call
ctx.messages.append(resp.choices[0].message)
if resp.choices[0].message.tool_calls:
for call in resp.choices[0].message.tool_calls:
tool_start = time.monotonic()
result = await execute_tool(call, ctx)
ctx.tool_latency += time.monotonic() - tool_start
ctx.step_latency.append(time.monotonic() - start)The loop hides the real cost: execute_tool may hit a rate limit and sleep for 20 seconds. Your benchmark must capture that.
Retries and fallback
Providers fail. When a model endpoint returns 429, the agent either retries or switches models. An OpenAI-compatible gateway such as n4n.ai that automatically falls back across 240+ models when a provider is degraded can reduce task failures, but the fallback handshake adds measurable seconds to a step—include it in your long-horizon agent latency benchmark.
If you hard-code a single provider, a 10-minute regional outage turns your task into a failure. The benchmark should inject outages to see how the orchestration copes.
Designing a long-horizon agent latency benchmark
Workload definition
Pick a task with a natural completion signal: a file written, a JSON schema validated, or a human-rated score above threshold. Synthetic “answer this question” loops underestimate variance because they lack tool dependencies.
We use a fixed corpus of 12 research queries, each requiring at least 20 tool calls. The agent must produce a 500-word memo with citations. That defines one task iteration. This workload is repeatable and exposes scheduling variance.
Metrics that matter
Record per-task wall clock (p50, p95, p99). Break down by phase:
- Model inference time (sum of chat completions)
- Tool execution time (external APIs)
- Idle/backoff time (sleeps, queues)
- Orchestration overhead (serialization, routing)
A task that finishes in 240s but spends 180s in backoff is a reliability problem, not a compute problem. The long-horizon agent latency benchmark must surface this split or it is useless.
{
"task_id": "eu-ai-act-03",
"p95_total_s": 312,
"phases": {
"inference_s": 54,
"tools_s": 121,
"backoff_s": 112,
"orch_s": 25
},
"steps": 28,
"fallbacks": 2
}Injecting failure
Run the benchmark with a proxy that drops 5% of requests and adds 2s latency to 10% of tool responses. Compare against a clean run. If p95 blows up by 3x, your agent is not production-ready.
# toxiproxy example
toxiproxy-cli toxic add -n latency -t latency -a latency=2000 -a jitter=500 my_proxy
toxiproxy-cli toxic add -n drop -t limit_data -a bytes=1 my_proxyChaos at the tool layer is as important as model layer. A research agent dependent on a flaky search API will show worse latency degradation than one with a cached corpus.
Implementation sketch
You can build this with asyncio and a single client pointed at any OpenAI-compatible endpoint. The key is wrapping every I/O with timers and a context object that survives fallbacks.
import asyncio, time, openai
client = openai.AsyncOpenAI(base_url="https://api.n4n.ai/v1", api_key="KEY")
async def benchmark_task(query, max_steps=30):
ctx = {"messages": [{"role": "user", "content": query}], "tool_latency": 0.0}
t0 = time.monotonic()
for _ in range(max_steps):
try:
r = await client.chat.completions.create(
model="auto",
messages=ctx["messages"],
tools=TOOL_SPECS,
)
# ... handle tools, append results
except openai.RateLimitError:
await asyncio.sleep(5) # simplistic; real code uses backoff
return time.monotonic() - t0Note the model="auto" directive: a gateway that honors client routing can pick the least-loaded model. That is a lever your benchmark should toggle. The same code runs against a direct provider by swapping the base URL.
Wrap tool execution with the same timing discipline. Log every sleep. The long-horizon agent latency benchmark lives or dies on instrumentation quality.
Tradeoffs in optimization
Model tiering
Using a frontier model for every step wastes latency and money. A common pattern: small model for tool selection, large model for synthesis. This cuts inference phase but adds a second call per step. Measure both. In our loops, tiering dropped p95 inference from 90s to 40s but added 10s orchestration.
Caching and state
Prompt caching reduces repeat context costs. Forward provider cache-control hints; a gateway that honors them stores the system prompt and tool specs. For research agents, the retrieved corpus changes per query, so cache hits are partial. Still, caching the planner prompt across steps saves real seconds.
client.chat.completions.create(
model="auto",
messages=ctx["messages"],
extra_headers={"cache-control": "ephemeral"}
)Concurrency control
Running 50 research tasks in parallel saturates your search API before the model. A long-horizon agent latency benchmark must include concurrency levels: 1, 10, 50. You will see p95 rise non-linearly. Implement a semaphore around tool calls, not just model calls.
tool_sem = asyncio.Semaphore(8)
async def execute_tool(call, ctx):
async with tool_sem:
return await actual_tool(call)Observability
Without distributed tracing, you will guess which phase hurts. Emit spans for each step, tool, and fallback. A per-token usage meter helps attribute cost but not latency; pair it with wall-clock spans.
Honest weighing of approaches
Fallback gateways reduce task failure but obscure per-provider latency. If you always measure behind an aggregator, you cannot tell which provider is slow. Solution: run the benchmark in two modes—direct to a provider, and through the gateway. The delta is the cost of resilience.
Smaller models reduce inference time but increase step count due to weaker planning. We observed a 7B model taking 45 steps where a 70B took 28. Total wall clock was equal; token cost halved. That is a tradeoff only visible in a long-horizon agent latency benchmark, not in a chat eval.
Caching helps but complicates correctness. Stale retrieved context can send the agent down wrong paths, increasing steps. Measure cache hit rate alongside latency to avoid hidden regressions.
Decisive takeaway
Stop reporting model TTFT as agent performance. Build a benchmark that drives full research tasks to completion under injected faults, records p95 wall-clock broken down by phase, and runs at production concurrency. Optimize the phase that dominates your numbers—usually tool latency or backoff, not inference. Only then will your agent survive contact with real users.