Agent loops have a nasty cost curve: every iteration replays the running history, the full tool schema, and whatever the last tool call returned, then appends more of the same before calling the model again. A loop that feels cheap on turn one can be paying for 10x the tokens by turn fifteen — not because the task got harder, but because the context never stops growing. Here are seven concrete techniques to shrink token usage in agent loops, roughly ordered from “fix this first” to “fix this once you’re optimizing in earnest.”
1. Trim tool schemas before they hit the model
Most agent frameworks load the entire toolset into every request’s function-calling schema, even when a given turn only needs two of your fifteen tools. A single tool definition with a detailed description and nested parameter schema easily runs 150-400 tokens; multiply that by a real toolset and you’re paying 3,000-6,000 tokens per call just to describe capabilities the model won’t use this turn.
Fix it with a lightweight router: classify the current step’s intent first (cheaply — see technique 6), then pass only the relevant subset of tool schemas into the actual reasoning call.
// instead of shipping all 15 tools every turn, resolve intent first,
// then send only what's relevant:
{
"tools": [
{ "name": "search_docs", "description": "..." },
{ "name": "read_file", "description": "..." }
]
}2. Replace transcript replay with rolling summarization
Chat-style agent loops that append every turn to a growing messages array are the single biggest source of quadratic token growth — turn 20 resends turns 1 through 19 in full. Instead, keep the last N turns verbatim (the model usually only needs recent context to act coherently) and collapse everything older into a running summary that’s regenerated every few turns.
if len(turns) > KEEP_RAW:
older = turns[:-KEEP_RAW]
state["summary"] = summarize(state.get("summary", ""), older)
turns = turns[-KEEP_RAW:]This bounds context growth to roughly O(1) per turn instead of O(n), and the summarization call itself can run on a small, cheap model since it’s compression, not reasoning.
3. Post-process tool outputs before they re-enter context
Search results, file reads, and API responses routinely come back 5-10x larger than what the model actually needs to act. A web search tool that returns full HTML snippets, or a file-read tool that returns an entire log file when only the last 20 lines matter, is quietly the most expensive part of the loop — because that bloated output gets appended to context and then replayed on every subsequent turn until it ages out.
Extract before you append: title + URL + one-line summary instead of full snippets; the relevant diff instead of the whole file; the top 5 rows instead of a 2,000-row CSV dump. Treat every tool’s return value as something to compress, not just something to pass through.
4. Offload working memory to a scratchpad instead of the prompt
Long-running agents (multi-file refactors, research tasks, multi-step form-filling) often try to carry all intermediate state — files touched, decisions made, partial results — directly in the prompt. That state doesn’t need to live in context at all. Write it to an external scratch file or key-value store, and give the agent a tool to read/update it. Re-inject only a short pointer or digest (“state saved to scratchpad/plan.md, 4 of 7 steps complete”) rather than the full state blob every turn. The model pulls in detail only when it actually needs it.
5. Cache the stable prefix
In most agent sessions, the system prompt, tool schemas, and few-shot examples are identical across every single call — only the tail end of the conversation changes. Without prompt caching, you pay full input-token price to re-process that stable prefix on every turn. With caching enabled (cache breakpoints supported by most major model providers today), that prefix is billed once and reused at a steep discount on subsequent calls, which matters most in exactly the tool-heavy, high-frequency loops this post is about.
If you’re calling models through a gateway rather than a single provider directly, check that caching directives pass through untouched — n4n.ai’s API is OpenAI-compatible and forwards cache-control hints to the underlying provider rather than stripping them, so this optimization survives the extra hop.
6. Tier the model per subtask, not per session
Not every step in an agent loop needs your most expensive model. Intent classification, tool selection, argument extraction, and “is this task done yet?” checks are usually simple enough for a small, cheap model — it’s only the final synthesis or multi-step reasoning step that benefits from a frontier model. Splitting a loop into “cheap router calls + expensive reasoning calls” often cuts blended cost far more than any single context-trimming trick, because those router-style calls tend to fire on every iteration.
This is also where routing infrastructure earns its keep: a gateway that can address 240+ models from a single OpenAI-compatible endpoint, with automatic fallback if a smaller model is degraded or rate-limited, makes it practical to actually run this tiering in production instead of hand-wiring three separate SDKs. Pay-per-token metering across the tiered calls also makes the savings visible immediately rather than buried in a monthly invoice.
7. Cap loop iterations and add explicit stop conditions
The most expensive token sink in an agent loop is often not any single call — it’s a loop that doesn’t know when to stop. Re-planning after a failed tool call, retrying the same search with slightly different wording, or looping between two states without making progress all mean re-sending the entire accumulated context, over and over, for no forward progress.
Guard against it explicitly: hard-cap iteration count, add a “done” check the agent must satisfy before exiting, and dedupe repeated tool calls (don’t let the model call the identical search twice because it forgot it already has the result). These guardrails cost almost nothing to implement and cap your worst-case token bill outright.
Where each technique saves you
| Technique | Saves tokens by | Effort |
|---|---|---|
| Trim tool schemas | Reducing per-call definition overhead | Low |
| Rolling summarization | Bounding transcript growth | Medium |
| Prune tool outputs | Shrinking what re-enters context | Low |
| Scratchpad offload | Removing state from the prompt entirely | Medium |
| Prefix caching | Discounting repeated stable context | Low |
| Model tiering | Cutting cost per call, not just size | Medium |
| Loop caps + stop conditions | Bounding worst-case iterations | Low |
None of these require rearchitecting your agent. Most are a few lines in the loop’s call-preparation step, and the highest-leverage ones — trimming tool schemas, pruning tool outputs, and capping iterations — are usually an afternoon’s work with an immediate, measurable drop in tokens per completed task.