Most teams instrument their agent’s total runtime but treat the loop as a black box. Agent planning overhead latency—the time spent generating reasoning, task decomposition, and next-action decisions—regularly accounts for the majority of wall-clock time in tool-heavy autonomous loops, yet it is rarely measured in isolation. If you optimize the wrong phase, you ship changes that do nothing.
Why planning overhead gets ignored
Engineers reach for time.perf_counter() around the whole run_agent() call and call it a day. That metric conflates two fundamentally different cost centers: the cognitive tax of deciding what to do, and the mechanical tax of doing it.
I have seen a production agent where the team blamed the vector database for 2-second steps. Profiling showed the DB call was 12 ms; the remaining 1.98 seconds were agent planning overhead latency from re-embedding the entire conversation history each step and re-serializing a growing scratchpad.
Most observability stacks trace the outer agent.run() and label it “inference” because an LLM is involved. That label is wrong. The LLM call may be 800 ms, but the agent also spends 200 ms serializing state, 100 ms retrieving vectors, and 50 ms parsing JSON. All of that is planning tax.
Anatomy of an agent loop step
A typical ReAct-style loop iterates through three phases per step:
- Context assembly – gather prior observations, system prompt, scratchpad.
- Inference – the model generates thoughts and a structured action.
- Action execution – call the tool, wait, format result.
Phases 1 and 2 are pure planning. Phase 3 is execution. Agent planning overhead latency is the sum of phases 1 and 2, excluding any time the process spends blocked on I/O that is not model-related.
A naive loop makes the planning cost invisible and quadratic:
def naive_loop(goal, max_steps=10):
state = init_state(goal)
for i in range(max_steps):
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=state.messages # grows every step
)
action = parse(resp)
obs = tool(action)
state.append(obs)
return stateEvery iteration resends the full message list. The prefill cost scales with cumulative context, so planning overhead grows even when the task does not.
What counts as planning
Any token generated before the agent commits to a tool call is planning. This includes:
- System prompt evaluation (if recomputed each step)
- Retrieval of prior steps from memory
- The model’s completion delay until it emits a stop token or function call
What counts as execution
Once the agent yields a function call, the clock for planning stops. The time to HTTP round-trip to a search API, parse its response, and validate the schema is execution. If the agent then reflects on the result, that reflection is a new planning phase.
Measuring agent planning overhead latency correctly
You need fine-grained spans. Wrap the model call and the surrounding context build separately from the tool invocation. Below is a minimal Python pattern using standard instrumentation.
import time
from contextlib import contextmanager
PLANNING_SPANS = []
EXECUTION_SPANS = []
@contextmanager
def span(kind: str):
start = time.perf_counter()
yield
elapsed = time.perf_counter() - start
(PLANNING_SPANS if kind == "plan" else EXECUTION_SPANS).append(elapsed)
def agent_step(query, tools):
with span("plan"):
action = model_decide(query, tools) # blocks on inference + ctx build
with span("exec"):
result = run_tool(action)
return result
def report():
p = sum(PLANNING_SPANS)
e = sum(EXECUTION_SPANS)
print(f"planning: {p:.2f}s exec: {e:.2f}s overhead%: {p/(p+e):.1%}")If you use OpenTelemetry, replace the lists with real spans:
from opentelemetry import trace
tracer = trace.get_tracer("agent")
with tracer.start_as_current_span("plan"):
resp = llm_call(...)
with tracer.start_as_current_span("exec"):
obs = tool_call(...)This separates agent planning overhead latency from execution latency at the granularity that matters.
A representative trace
Consider a five-step agent solving a data lookup task. The tool is a fast local SQLite query (sub-50ms). The model is a 70B-class endpoint behind an OpenAI-compatible gateway.
{
"steps": [
{"step": 1, "plan_ms": 820, "exec_ms": 42},
{"step": 2, "plan_ms": 910, "exec_ms": 38},
{"step": 3, "plan_ms": 1150, "exec_ms": 51},
{"step": 4, "plan_ms": 780, "exec_ms": 44},
{"step": 5, "plan_ms": 1020, "exec_ms": 40}
]
}Total planning: 4.68s. Total execution: 0.215s. Agent planning overhead latency is 95.6% of the loop. Shipping a 10x faster database changes nothing perceptible.
Contrast with a loop that calls a slow external API:
{
"scenario": "slow_external_api",
"steps": [
{"plan_ms": 900, "exec_ms": 3200},
{"plan_ms": 850, "exec_ms": 3100}
]
}Here planning is roughly 35%—still real money and time, but not the dominant factor. You tune differently in each regime.
Tradeoffs: more planning vs less
Deliberative agents plan extensively to reduce tool errors. A coding agent that writes a full test plan before editing files will incur high agent planning overhead latency but may need only one execution attempt. A reactive agent that guesses and observes may execute ten times but spend less time per planning phase.
The math is rarely linear. If planning takes 1s and prevents one 5s erroneous tool call, you win. If planning takes 3s and merely rephrases the same action, you lose.
You must measure both the overhead and the error correction rate. A simple ablation: disable reflection and measure task success vs total latency.
# run agent with reflection disabled
PYTHONPATH=. AGENT_REFLECT=0 python bench.py --task suite_a
# run with forced plan-before-act
AGENT_REFLECT=1 python bench.py --task suite_aCompare the planning percentages and success rates. The crossover point defines your optimal planning budget. You can enforce it dynamically:
def plan_with_budget(budget_ms, ctx):
start = time.perf_counter()
while time.perf_counter() - start < budget_ms:
candidate = model_propose(ctx)
if validator(candidate):
return candidate
return fallback(ctx)Where the inference layer compounds the cost
Planning tokens are repetitive. Each step resends the system prompt, prior thoughts, and tool schemas. Provider-side prompt caching cuts this dramatically, but only if your client forwards the right cache-control hints. An inference gateway that honors routing directives and forwards cache-control can drop planning latency by reusing cached prefix computations across steps.
For example, n4n.ai exposes an OpenAI-compatible endpoint that addresses 240+ models and forwards provider cache-control hints, so a multi-step agent reusing the same system prefix avoids recomputing it on every planning call. That directly reduces agent planning overhead latency without changing agent logic.
If your gateway does not support cache hints, you pay full prefill cost per step. That is pure tax.
Honest limitations of the metric
Planning overhead is not uniformly bad. In agents that call rate-limited external APIs, execution dominates because you are blocked on someone else. In those loops, measuring agent planning overhead latency is still useful to know how much headroom you have to add safety checks without hurting UX.
Also, overlapping planning and execution via speculative actions violates the clean separation. If you fire a tool before the model finishes reasoning (rare but possible in streaming architectures), the spans overlap and you need concurrent timers. Likewise, if you batch multiple model calls in parallel for candidate plans, the simple sequential span model undercounts.
Decisive takeaway
Instrument planning and execution as distinct spans from day one. Agent planning overhead latency is the dominant cost in most LLM agent loops that are not bottlenecked on slow tools, and you cannot optimize what you have merged into a single timer. Measure it, cache your prefixes at the gateway level, and tune the planning budget against task success—not against a vague sense that “the agent is slow.” Build the measurement first; the optimization is then a matter of reading the chart.