Agents that invoke search, database, or code-execution tools routinely receive payloads far larger than the space left in the prompt. This tool output context overflow either truncates the model’s reasoning, spikes token spend, or returns a hard 400 from the provider. You fix it with engineering constraints at the tool boundary, not by hoping the next model has a bigger window.
1. Cap output at the tool boundary
The first line of defense is a hard limit enforced the moment a tool returns. Do not wait for the agent loop to assemble the prompt and discover it is too large. Wrap every tool call in a decorator or middleware that measures size and either truncates with a clear marker or returns a structured error the agent can handle.
def cap_tool_output(max_chars: int = 6000):
def decorator(fn):
def wrapper(*args, **kwargs):
raw = fn(*args, **kwargs)
if isinstance(raw, str) and len(raw) > max_chars:
# 6000 chars ~ 1500 tokens for English; tune per model
return raw[:max_chars] + f"\n...[truncated, {len(raw)} chars total]"
return raw
return wrapper
return decorator
@cap_tool_output(6000)
def search_web(query: str) -> str:
# hypothetical HTTP call
return http.get("/search", params={"q": query}).textThe pitfall here is naive character truncation. If your tool returns a list of JSON objects, slicing the string mid-object produces invalid JSON and loses the final row silently. Prefer capping at the row or element level:
def cap_rows(raw_list, max_rows=20):
if len(raw_list) > max_rows:
return raw_list[:max_rows], len(raw_list) - max_rows
return raw_list, 0Tradeoff: you may drop relevant data. Mitigate by sorting or filtering at the source query before returning.
2. Bind tools to narrow output schemas
Most tool output context overflow is self-inflicted because tools return everything they have. Define an explicit return contract and reject anything broader.
{
"tool": "order_status",
"returns": {
"order_id": "string",
"state": "enum[paid,shipped,delivered]",
"eta": "iso_date|null"
}
}If the agent later needs line items, it makes a second call with a specific order_id. This keeps the average tool payload under a few hundred tokens. The cost is an extra round-trip, but that latency is predictable and small compared to stuffing 10 KB of HTML into the context.
A common mistake is returning human-readable HTML or markdown “for debugging.” That text is poison for context windows. Emit machine schema to the agent, and keep verbose logs in your observability pipeline instead.
3. Compress with a side model when raw data is unavoidable
Some tools—like a code interpreter dumping a stack trace or a PDF parser—cannot be shrunk by schema alone. Offload compression to a smaller, cheaper model before the data enters the main agent context.
from openai import OpenAI
# OpenAI-compatible client pointed at a gateway
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key=KEY)
def compress_tool_output(raw: str, budget_tokens: int = 300) -> str:
resp = client.chat.completions.create(
model="anthropic/claude-3-haiku",
messages=[{
"role": "user",
"content": f"Extract only facts an agent needs to act. Raw:\n{raw}"
}],
max_tokens=budget_tokens,
)
return resp.choices[0].message.contentRouting the summarization through an inference gateway such as n4n.ai lets you pin a cheap model via a routing directive and still get automatic fallback if that provider is rate-limited, so the compression step never becomes a single point of failure. The tradeoff is added latency (often 200–500 ms) and a second billable call. Only compress when the raw payload exceeds your cap; pass through small outputs directly.
4. Use tiered retrieval instead of full dumps
Design tools as two-stage operations. The first call returns pointers (IDs, offsets, URLs). The agent decides what to expand.
def list_tickets(user_id: str) -> list[dict]:
return [{"id": t.id, "subject": t.subject[:50]} for t in db.query(user_id)]
def get_ticket(ticket_id: str) -> dict:
return db.get_full(ticket_id) # only called on demandThis pattern keeps the initial context lean and defers the tool output context overflow risk to a targeted fetch. The agent’s planning ability improves because it sees the shape of available data before committing tokens to details.
Pitfall: agents sometimes blindly expand all pointers in a loop. Add a per-step expansion limit (e.g., max 3 tickets per reasoning step) to prevent a cascade.
5. Evict and checkpoint older tool outputs
Once the agent has acted on a tool result, that raw text is dead weight. Implement a context manager that keeps the last N tool outputs verbatim and summarizes older ones into a rolling checkpoint.
class ContextWindow:
def __init__(self, keep: int = 4):
self.recent = []
self.checkpoint = ""
self.keep = keep
def add(self, tool_out: str):
self.recent.append(tool_out)
while len(self.recent) > self.keep:
old = self.recent.pop(0)
self.checkpoint += f" [summary:{old[:80]}...]"Feed the model checkpoint + recent instead of the full history. You lose fine-grained recall of old outputs, but most agent tasks are myopic. If long-term memory is required, write the full output to external storage and give the agent a recall tool.
6. Meter tokens per tool call
You cannot manage tool output context overflow without measurement. Capture usage.completion_tokens and prompt_tokens on every model call and attribute them to the tool that contributed the bulk of the prompt. A gateway with per-token usage metering gives you line-item cost per route; even without that, log the estimated token count of each tool payload before insertion.
Set an alert when a single tool output exceeds 25% of your model’s context. That threshold catches overflow before it happens and exposes poorly behaved tools during development.
7. Fuzz your tool interfaces in CI
The fastest way to learn your limits are wrong is a production incident. In tests, send deliberately oversized, malformed, and deeply nested outputs through every tool.
def test_search_overflow():
huge = "x" * 100_000
out = search_web(huge) # mock transport
assert "[truncated" in outRun this in CI so regressions in tool serialization surface immediately. Include random Unicode and partial JSON to verify your parser fails safe.
Common pitfalls and tradeoffs
- Counting only the tool output. System prompts, few-shot examples, and prior turns consume context too. Budget for them explicitly.
- Over-compressing. A haiku model summary can drop the one field the agent needed. Validate compressed output against a schema before trusting it.
- Silent truncation. Never truncate without a marker the model can see. “…[truncated]” lets the agent decide to refetch.
- Latency vs. safety. Tiered retrieval adds round-trips but is the only scalable fix for large corpora.
Tool output context overflow is a systems problem, not a model problem. Enforce caps, bind schemas, compress selectively, tier access, evict aggressively, and measure constantly. Do that and your agent will stay within the window even as tools grow messier.