Benchmarking agent latency providers is not the same as timing a single chat completion. An agent runs a loop: model call, tool execution, response parsing, sometimes retries, and each provider behaves differently under load. To benchmark agent latency providers fairly, you need a fixed task, instrumented loops, and enough trials to see tail latency instead of best-case numbers.
Step 1: Pin a realistic agent task
Pick one task that forces the agent through at least one tool call and a final answer. If you test with a bare “hello”, you measure only time-to-first-token on a trivial generation. Use a task that exercises function calling and multi-step reasoning.
A good baseline: ask the model to compute a math expression using a supplied calc tool. The expression should be non-trivial enough that the model cannot answer from pretraining alone with high confidence, but deterministic for verification.
TASK_PROMPT = "Compute (23*17)+sqrt(42). You must use the calc tool to evaluate the expression."Keep the system prompt short and identical across providers. Any difference in prompt length or wording will contaminate your latency comparison.
Step 2: Instrument the agent loop
Write a minimal agent loop that records wall-clock time for each LLM call and each tool execution separately. Do not use time.time(); use time.perf_counter() for sub-millisecond resolution.
import time, json
from openai import OpenAI
def run_agent(client, model, prompt, max_steps=5):
messages = [{"role": "user", "content": prompt}]
tool = {
"type": "function",
"function": {
"name": "calc",
"description": "Evaluate a math expression",
"parameters": {
"type": "object",
"properties": {"expr": {"type": "string"}},
"required": ["expr"]
}
}
}
timings = []
for step in range(max_steps):
t0 = time.perf_counter()
resp = client.chat.completions.create(
model=model,
messages=messages,
tools=[tool],
tool_choice="auto"
)
t1 = time.perf_counter()
msg = resp.choices[0].message
messages.append(msg)
if not msg.tool_calls:
timings.append({"step": step, "llm_ms": (t1 - t0) * 1000, "tool_ms": 0.0})
return {"answer": msg.content, "timings": timings, "steps": step + 1}
t2 = time.perf_counter()
expr = json.loads(msg.tool_calls[0].function.arguments)["expr"]
result = eval(expr) # demo only; use a safe parser in production
t3 = time.perf_counter()
messages.append({
"role": "tool",
"tool_call_id": msg.tool_calls[0].id,
"content": str(result)
})
timings.append({
"step": step,
"llm_ms": (t1 - t0) * 1000,
"tool_ms": (t3 - t2) * 1000
})
return {"answer": None, "timings": timings, "steps": max_steps}This separates model latency from local tool latency. If you only record total runtime, a slow local SQL call will mask provider differences.
Step 3: Configure provider endpoints
Use an OpenAI-compatible client for every provider you can. For providers without a native OpenAI schema, put a proxy or gateway in front. A gateway like n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models, so you can benchmark agent latency providers by changing the model string instead of rewriting HTTP clients.
import os
from openai import OpenAI
providers = {
"openai": {
"base_url": "https://api.openai.com/v1",
"api_key": os.environ["OPENAI_KEY"],
"model": "gpt-4o-mini"
},
"anthropic_via_gateway": {
"base_url": "https://api.n4n.ai/v1",
"api_key": os.environ["N4N_KEY"],
"model": "anthropic/claude-3-5-sonnet"
},
"local_vllm": {
"base_url": "http://localhost:8000/v1",
"api_key": "empty",
"model": "mistralai/Mistral-7B-Instruct-v0.3"
}
}Set base_url and model per entry. Keep max_tokens and sampling params (temperature=0) fixed.
Step 4: Run enough trials to capture tail latency
A single run is noise. Run at least 20 cold trials per provider, discard the first two as warmup, and record per-run totals. If your production agent runs concurrently, simulate that with asyncio or threading.
import statistics, time
def benchmark(providers, trials=25):
results = {}
for name, cfg in providers.items():
client = OpenAI(base_url=cfg["base_url"], api_key=cfg["api_key"])
runs = []
for i in range(trials):
start = time.perf_counter()
out = run_agent(client, cfg["model"], TASK_PROMPT)
end = time.perf_counter()
runs.append({
"total_ms": (end - start) * 1000,
"steps": out["steps"],
"llm_ms": sum(t["llm_ms"] for t in out["timings"]),
"tool_ms": sum(t["tool_ms"] for t in out["timings"]),
"ok": out["answer"] is not None
})
results[name] = runs[2:] # drop warmup
return resultsRun this from the same region you deploy in. Cross-continent latency will dominate any provider-internal speed difference.
Step 5: Aggregate and compare
Compute median (p50) and 95th percentile (p95) total latency. Also track error rate and step count—some providers loop longer before answering.
def summarize(results):
for name, runs in results.items():
totals = [r["total_ms"] for r in runs]
p50 = statistics.median(totals)
p95 = sorted(totals)[int(len(totals) * 0.95)]
err = sum(1 for r in runs if not r["ok"])
print(f"{name}: p50={p50:.0f}ms p95={p95:.0f}ms "
f"errors={err}/{len(runs)} steps={runs[0]['steps']}")When you benchmark agent latency providers at scale, p95 matters more than p50. An agent that usually answers in 2s but spikes to 12s under provider congestion will break interactive UX.
Plot the distributions. A box plot per provider reveals outliers that averages hide. If one provider shows a bimodal spread, it is likely hitting a rate limit and silently retrying.
Step 6: Control for variables that skew results
Several factors distort raw numbers:
- Time of day: Provider load varies. Run all providers interleaved, not sequentially across days.
- Caching: Repeated identical prefixes can hit provider prompt caches. n4n.ai forwards provider cache-control hints, which matters when your agent re-sends long system prompts each step. Test with and without cache warm.
- Fallback paths: If a gateway auto-fails over to a secondary provider on 429s, your latency includes the failed call plus the retry. Measure with fallback enabled if that is your production config.
- Streaming: Non-streaming waits for full generation. Agents often use streaming to start tool parsing early. Switch
stream=Trueand measure time-to-tool-call instead of time-to-final-token.
Adjust your run_agent to support streaming if that matches production:
resp = client.chat.completions.create(
model=model, messages=messages, tools=[tool],
stream=True
)
# accumulate delta, record t_first_tool when tool_call delta appearsVerify success
You have a working benchmark when you can produce a table like this from a single script run:
openai: p50=1840ms p95=3100ms errors=0/23 steps=2
anthropic_via_gateway: p50=2100ms p95=4200ms errors=1/23 steps=2
local_vllm: p50=900ms p95=1500ms errors=0/23 steps=3Success means: (1) the task completes correctly for every provider, (2) you have per-phase timings, (3) p50/p95 are stable across repeated benchmark invocations, and (4) you can swap a model string and re-run without code changes. At that point you can make a defensible decision about which provider meets your agent’s latency budget.