Token costs of agentic loops and how to control them

Agentic loop token costs are the silent budget killer in production LLM systems. Every iteration appends observations, tool outputs, and reasoning to the context, and most teams don’t notice the linear (or worse) growth until the bill arrives. This guide gives an ordered path to measure and cap those costs without crippling agent capability.

1. Instrument every loop iteration

You cannot control what you do not measure. Wrap each model call in your agent loop and log the usage object returned by the API. Every OpenAI-compatible endpoint returns prompt_tokens, completion_tokens, and total_tokens on each completion. The mistake most teams make is looking only at the final tally after the agent finishes, which hides where the spend actually happened.

from openai import OpenAI

client = OpenAI(base_url="https://api.your-gateway.com/v1", api_key="sk-...")

def agent_step(messages, tools, model="gpt-4o"):
    resp = client.chat.completions.create(
        model=model,
        messages=messages,
        tools=tools,
    )
    u = resp.usage
    print(f"step prompt={u.prompt_tokens} completion={u.completion_tokens} total={u.total_tokens}")
    return resp

Run a representative task and record per-step tokens. The biggest driver of agentic loop token costs is the repeated system prompt plus accumulated history on every turn. If step 1 uses 2k tokens and step 6 uses 12k, you have a context growth problem, not a model problem. Break the log down by phase: planning, tool call, reflection. A planning step that re-reads the entire tool schema every time is a prime suspect.

Pitfall: trusting UI dashboards that show only daily totals. You need per-call granularity tied to a run ID to spot runaway loops.

2. Separate static from dynamic context

The system prompt, tool schemas, and few-shot examples rarely change within a run. Mark them as cacheable so the provider bills cached tokens at a discount and skips recomputation. With an OpenAI-compatible gateway that forwards cache hints, send cache control on the static block:

{
  "model": "anthropic/claude-3-5-sonnet",
  "messages": [
    {"role": "system", "content": "You are a DevOps agent. Tools: kubectl, terraform, sql. Use them sequentially.", "cache_control": {"type": "ephemeral"}}
  ]
}

This keeps the invariant prefix out of the metered prompt token count on subsequent calls. Tradeoff: caches expire (often 5–60 minutes depending on provider). For long-running agents, re-warm the cache by repeating the static prefix periodically, or accept recomputation on cold starts.

Reducing agentic loop token costs is mostly about context discipline, and caching the invariant parts is the highest-leverage move you can make before touching model choice.

3. Cap tool output before it enters context

Tool responses are the primary source of unbounded growth. A SQL query returning 500 rows or a web fetch dumping HTML will inflate prompt tokens on the next call and every call after that. Truncate or structurally reduce before appending:

import json

def trim_json_result(payload: dict, max_keys=10) -> str:
    if len(payload.get("rows", [])) > max_keys:
        payload = {**payload, "rows": payload["rows"][:max_keys]}
        payload["truncated"] = True
    return json.dumps(payload)

messages.append({"role": "tool", "content": trim_json_result(huge_response)})

Better: extract only fields the agent needs via a cheap parser or a small model. Never feed raw REST responses into the loop. For text, use character budgets:

def trim_text(raw: str, max_chars=1500) -> str:
    return raw if len(raw) <= max_chars else raw[:max_chars] + "\n...[truncated]"

Pitfall: truncating blindly may drop the one row the agent needed. Use schema-aware extraction (e.g., JSON path) instead of naive slicing for structured data. Test the trimmer against historical runs to confirm the agent still succeeds.

4. Use an external scratchpad

The agent does not need its entire chain-of-thought in the LLM context. Keep verbose reasoning, intermediate JSON, and failed attempts in an external store (Redis, Postgres) keyed by run ID. Inject only a pointer or a two-line summary into the messages.

import redis
r = redis.Redis()

def log_to_scratchpad(run_id, trace):
    r.append(f"scratch:{run_id}", trace + "\n")

log_to_scratchpad(run_id, verbose_tool_trace)
messages.append({"role": "user", "content": f"Step {i} complete. Scratchpad key {run_id} holds details."})

This keeps the live context lean. The tradeoff is added engineering: you must build retrieval if the agent later needs to recall earlier details. A simple r.get(f"scratch:{run_id}") sliced to the last N bytes is often enough.

5. Enforce hard ceilings and route to cheaper models

Set max_tokens on completions and a maximum step count in your loop. Beyond that, halt or fall back to a smaller model for routing decisions. Use a clear routing policy:

MAX_STEPS = 12
BIG = "gpt-4o"
SMALL = "gpt-4o-mini"

for i in range(MAX_STEPS):
    model = SMALL if i % 4 != 0 else BIG  # use frontier model for planning only
    resp = agent_step(messages, tools, model=model)
    if resp.choices[0].finish_reason == "stop":
        break

A gateway such as n4n.ai honors client routing directives and provides automatic fallback when a provider is rate-limited, letting you enforce a cheap-model-first policy without writing custom retry logic. That directly caps agentic loop token costs during traffic spikes because a degraded expensive provider automatically yields to a cheaper one instead of blocking the loop.

Tradeoff: a small model may misroute and cause extra steps. Measure step count vs token savings on a labeled task set before shipping the policy.

6. Compress history with rolling summaries

After N steps, summarize the message history with a dedicated call and replace older messages with the summary. Use the smallest capable model for the summarizer.

def summarize_history(client, msgs):
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "system", "content": "Compress the agent trace into 3 bullet points."}] + msgs,
    )
    return resp.choices[0].message.content

if len(messages) > 20:
    summary = summarize_history(client, messages[1:10])
    messages = [messages[0]] + [{"role": "system", "content": "Summary: " + summary}] + messages[10:]

This bounds context growth to O(1) per cycle after warmup. Pitfall: summaries lose nuance. Keep the most recent two or three tool interactions unsummarized to preserve grounding. If the agent starts repeating actions, your summary is too aggressive.

7. Meter and alert per agent run

Pull usage from each call and accumulate per run ID. If your endpoint provides per-token metering, pipe it to your observability stack. Compute run cost by multiplying total tokens by your negotiated rate (or use the gateway’s usage export).

run_tokens = 0
for step_data in run_steps:
    run_tokens += step_data["total_tokens"]
if run_tokens > 50000:
    alert(f"Agent run {run_id} exceeded 50k tokens (approx ${run_tokens*0.00001:.2f})")

Set alerts on median cost per task, not just outliers. A slow creep in agentic loop token costs shows up as median drift before it shows up as a spike. Track cost per successful task, not per run, to avoid rewarding agents that fail fast and cheap.

Common pitfalls and tradeoffs

• Over-caching: relying on cache across runs with slightly different system prompts invalidates the cache and wastes money.
• Tiny context windows: aggressively trimming may cause the agent to repeat failed actions, increasing total steps and negating savings.
• Model downgrade without testing: swapping to a mini model for planning can multiply steps, erasing token gains.
• Summary drift: compressing too early loses the constraint that the agent was respecting, causing violations downstream.

Controlling agentic loop token costs is an engineering discipline, not a config flag. Measure per step, cache the static, truncate the dynamic, externalize the verbose, and cap the loop. Do that and your agent stays useful at a predictable price.