When you build an agentic system, the gap between agent latency vs model speed is where your users feel pain. A model that streams at 100 tokens per second means little if your agent makes eight sequential calls before returning a result.
The compounding math of agent steps
Agents are not single completions. They are graphs of reasoning, tool calls, and retries. Each node adds fixed overhead independent of how fast the model generates text.
Consider a typical ReAct loop: plan, call tool, observe, reflect, answer. If each LLM call takes 800 ms of round-trip latency plus generation, and you have five calls, you have at least 4 seconds before the user sees anything useful. Swap in a model that is 2x faster at token generation but still has the same 800 ms round trip, and you save maybe 300 ms total.
The arithmetic is brutal. Let L be per-step latency, S steps, G generation time. End-to-end latency ≈ S * L + G_total. Reducing G by 50% barely moves the needle when S * L dominates. Most production agents I’ve shipped have S between 3 and 12 for non-trivial tasks, so the fixed tax multiplies.
Where the milliseconds actually go
Time to first token and network overhead
Most engineers benchmark models by tokens/sec after the first token. That metric ignores TTFT (time to first token) and network hops. A request from your server to a provider, then to a gateway, then to the model host, adds 100–300 ms each way. If your agent runs in a loop, that tax is paid per step.
import time, openai
client = openai.OpenAI(base_url="https://api.example.com/v1", api_key="sk-...")
start = time.perf_counter()
stream = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Ping"}],
stream=True,
)
first = None
for chunk in stream:
if chunk.choices[0].delta.content:
first = time.perf_counter()
break
print(f"TTFT: {(first-start)*1000:.0f}ms")In practice, TTFT often exceeds generation time for short prompts. Agent latency vs model speed is therefore decided by infrastructure, not just weights.
Tool execution and context assembly
Between LLM calls, your code runs tools: SQL queries, vector searches, HTTP fetches. Those can take 200 ms to 2 s. You also rebuild the message list, re-embed, or truncate history. None of that is accelerated by a faster model.
# A typical agent step
def step(messages):
t0 = time.perf_counter()
resp = llm_chat(messages) # network + TTFT + gen
t1 = time.perf_counter()
tool_result = run_tool(resp.tool_call) # local or remote work
t2 = time.perf_counter()
messages.append(resp)
messages.append(tool_result)
print(f"llm {t1-t0:.2f}s tool {t2-t1:.2f}s")If tool consistently beats llm, optimizing the model is low leverage. Context assembly is worse: serializing 4k tokens of state, running a renderer, or calling an embedding model adds pure fixed cost. I’ve measured 150 ms just to reconstruct the prompt object in Python.
Faster models can create slower agents
The trap: a “fast” small model may lack the capability to solve the task in one or two steps. It loops, retries, and calls helpers, inflating step count.
Capability vs step count tradeoff
Suppose a task needs a complex SQL query. A 70B-class model writes it correctly first try (2 steps). A 7B model guesses, gets a schema error, retries, uses a describe-table tool, retries again (6 steps). Even if the 7B is 4x faster per token, the extra four round trips kill you.
| Model | Steps | Per-step latency | Total |
|---|---|---|---|
| Large | 2 | 1.2s | 2.4s |
| Small | 6 | 0.5s | 3.0s |
This is the core of agent latency vs model speed: the fastest agent is often the one that takes the fewest correct steps, not the one with the highest tokens/sec. When evaluating builds, always compare agent latency vs model speed under real step counts, not synthetic completions.
Measuring what matters: end-to-end agent latency
Instrument the whole trajectory. Don’t trust provider dashboards that show only generation time.
A minimal timing harness
Wrap your agent loop with span timers. Aggregate p50/p90 step counts and per-step latency.
import time, functools
def span(name):
def deco(f):
@functools.wraps(f)
def inner(*a, **k):
s = time.perf_counter()
out = f(*a, **k)
print(f"{name}: {(time.perf_counter()-s)*1000:.0f}ms")
return out
return inner
return deco
@span("plan")
def plan(m): return llm(m)
@span("tool")
def tool(r): return run_tool(r)
@span("reflect")
def reflect(m): return llm(m)Run this in staging with real tasks. You will see that variance comes from step count, not model throughput. Track p90 step count: if it spikes to 15 on edge cases, that dominates user complaints more than a 20% generation slowdown.
Routing and fallback: reducing variance, not mean
Provider degradation is real. When a model host throws 429s, your agent stalls. A gateway that offers automatic fallback to a healthy provider can keep per-step latency bounded. n4n.ai does this at the endpoint level, shielding your loop from a single vendor’s outage. But fallback does not reduce the number of steps; it only caps worst-case delay.
Honor client routing directives and forward cache-control hints to exploit provider prompt caches. That cuts TTFT on repeated context—a real win for agents that resend system prompts each step.
{
"model": "anthropic/claude-3.5-sonnet",
"messages": [{"role": "system", "content": "You are a terse agent."}],
"route": {"fallback": ["openai/gpt-4o", "meta/llama-3.1-70b"]},
"cache_control": {"type": "ephemeral"}
}The takeaway: treat latency as a property of the graph, not the node. Fallback is insurance, not acceleration.
Tradeoffs you should accept
- Use smaller models for classification or routing steps where step count stays fixed and the task is easy. A 7B judge that runs in 200 ms beats a 70B judge if it’s right 95% of the time.
- Use large models for steps that determine whether the agent terminates early. A planning call that cuts three later steps pays for itself at any speed.
- Cache aggressively; re-sending 2k tokens of system prompt every step is pure waste. Use provider cache-control or local prefix caching.
- Batch independent tool calls concurrently. Latency is parallelizable when the dependency graph allows.
import asyncio
async def parallel_tools(calls):
return await asyncio.gather(*[run_tool_async(c) for c in calls])If your agent fetches three docs and queries a DB with no interdependency, fire them together. That converts 3 * L_tool into max(L_tool).
Decisive takeaway
Optimize agent latency vs model speed by minimizing sequential steps and fixed overhead before chasing token throughput. Profile end-to-end, pick models by step-efficiency, and use fallback only to bound tail latency. A 20% faster model is a rounding error; a 30% reduction in steps is a product win. Build the agent graph first, then choose the models to fit the nodes.