Agent loop latency: how context growth slows each step

Agent loop context growth latency is the silent tax that breaks latency budgets in production agentic systems. Each iteration appends tool results, observations, and intermediate reasoning to the prompt, so the token count climbs monotonically—and the cost per step rises with it. Engineers who model agent latency as constant per hop are surprised when the tenth step takes three times longer than the first.

The mechanics of a growing context

A typical ReAct-style loop keeps the entire conversation history in memory and feeds it back to the model every step:

messages = [{"role": "system", "content": SYSTEM_PROMPT}]
while not done:
    resp = client.chat.completions.create(model="gpt-4o", messages=messages)
    msg = resp.choices[0].message
    messages.append(msg)
    tool_call = parse_tool(msg)
    if tool_call:
        result = run_tool(tool_call)
        messages.append({"role": "tool", "content": result})

The messages list is the context. On step one it might be 1,200 tokens. After five tool calls with verbose JSON returns, it is 9,000. After twenty steps, it can exceed 30,000. The model sees all of it every time because the API is stateless—unless you explicitly manage caching.

This is where agent loop context growth latency starts to bite. The loop does not just do more work; each unit of work gets slower because the input grows.

Why latency grows faster than you expect

Prefill cost dominates at scale

Transformers compute attention over the full input sequence before generating the first token. Even with FlashAttention and fused kernels, prefill time grows roughly quadratically with sequence length on most hardware, and at best sub-quadratically with heavy optimization. In practical terms: prefilling 4K tokens might take ~150–300 ms on a capable GPU endpoint; prefilling 32K tokens commonly takes 1–3 seconds. That delay is pure dead time before the user sees any output.

Decode cost is context-sensitive

Once prefill completes, the model generates tokens autoregressively. Each generated token requires a read of the KV cache for all prior tokens. The cache itself is large at long context, and memory bandwidth becomes the bottleneck. A step that emits 200 tokens against an 8K context is cheaper per token than the same step against a 40K context, even if the generated length is identical.

Network and payload overhead

Larger requests mean larger HTTP bodies. If you send 30K tokens as JSON on every step, you are uploading ~120 KB each time. That is not huge, but it adds measurable round-trip time on constrained links and increases serialization cost in your worker.

Measuring agent loop context growth latency

You cannot tune what you do not measure. Build a minimal harness that records tokens and wall-clock time per step:

import time, tiktoken
enc = tiktoken.get_encoding("o200k_base")

def step_latency(client, model, messages):
    t0 = time.time()
    resp = client.chat.completions.create(model=model, messages=messages)
    elapsed = time.time() - t0
    n_tok = sum(len(enc.encode(m.get("content", ""))) for m in messages)
    return resp, elapsed, n_tok

# log (step_index, n_tok, elapsed) and plot

Run this against your real tool outputs. The curve will not be flat. The first step might be 400 ms; the fifteenth might be 2.5 s. That delta is your agent loop context growth latency penalty, and it is reproducible.

Mitigation strategies that actually work

Trim tool outputs at the edge

Most tools return more than the model needs. A SQL query result with 500 rows can be reduced to “10 rows shown, 490 truncated” with a note. Wrap tool execution:

def bounded_result(raw, max_chars=2000):
    if len(raw) <= max_chars:
        return raw
    return raw[:max_chars] + f"\n... [{len(raw)-max_chars} chars truncated]"

This caps the marginal growth per step. You trade occasional loss of detail for predictable latency.

Periodic compaction

Every N steps, summarize the conversation history and replace the old messages with a single compact message. This is a deliberate information bottleneck:

if step % 8 == 0 and step > 0:
    summary = summarize(client, messages)
    messages = [messages[0], {"role": "system", "content": "Summary: " + summary}]

The risk is lost nuance, but for long tasks the alternative is timeout or cost blowup.

Prefix caching across loop iterations

The system prompt and any static instructions do not change. If your provider supports cache control, mark that prefix so the prefill is computed once and reused. An OpenAI-compatible gateway such as n4n.ai forwards these cache-control hints to the upstream provider, so the stable system prefix is prefilled once and reused across loop steps.

{
  "model": "anthropic.claude-3-5-sonnet",
  "messages": [
    {"role": "system", "content": "You are a strict agent.", "cache_control": {"type": "ephemeral"}}
  ]
}

This does not shrink the growing dynamic context, but it removes the repeated cost of the fixed part.

Route smaller models for internal steps

Not every step needs a frontier model. Use a cheap model to decide whether a tool call is even necessary, or to parse tool output into a structured form. Reserve the large context model for synthesis. This hybrid loop keeps the heavy context on fewer steps.

Tradeoffs and failure modes

Compaction can erase a critical constraint mentioned early (“never delete the user record”). Sliding windows drop the oldest context, which may contain the original user intent. Truncation can hide a key error string from a tool. Each mitigation is a deliberate information loss; you must verify it against your task success rate, not just latency.

Prefix caching helps only if the prefix is truly stable. If you inject a timestamp or request ID into the system prompt every call, the cache breaks. Keep the cached prefix immutable.

Smaller models introduce inconsistency. You now maintain two prompt formats and two failure modes. The latency win is real, but the debugging cost is higher.

A decisive takeaway

Treat context as a per-loop budget, not an unbounded buffer. Measure agent loop context growth latency on your real workload before shipping, set a hard token ceiling (e.g., 24K), and enforce it with trimming or compaction. Use provider cache control for the static prefix, and route only the steps that need full reasoning to the large model. Agents that ignore context growth will appear snappy in a demo and fall over in a long session; engineers who budget context ship systems that stay fast at step fifty.