8 ways to reduce token spend in production agents

Production agents burn tokens in silent ways: oversized system prompts, verbose tool results, and unnecessary model upgrades. To reduce token spend agents incur at scale, you need structural changes to how context is built and how calls are routed—not just a smaller temperature.

1. Cache static system prompts and reuse them across turns

Most agents send the same 1k–3k token system prompt on every request. Provider prompt caching turns that repeated prefix into a cached hit, but you must mark it explicitly for Anthropic or rely on automatic caching with OpenAI.

Use cache control on the stable prefix. In Anthropic’s native SDK:

import anthropic
client = anthropic.Anthropic()
resp = client.messages.create(
    model="claude-3-5-sonnet",
    system=[{"type": "text", "text": SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"}}],
    messages=[{"role": "user", "content": user_msg}]
)

For OpenAI, prefix caching triggers automatically on prompts longer than 1024 tokens; keep the static part identical byte-for-byte. Never append a timestamp or request ID to the system block or you invalidate the cache.

2. Route subtasks to smaller models

A common mistake is using a frontier model for intent classification or JSON reformatting. To reduce token spend agents pay per step, split the pipeline: use Haiku or GPT-4o-mini for extraction, reserve Opus for reasoning.

Implement a simple router in code:

def pick_model(task: str) -> str:
    if task in ("classify", "extract_entities"):
        return "gpt-4o-mini"
    if task == "complex_plan":
        return "claude-3-opus"
    return "gpt-4o"

A gateway like n4n.ai exposes one OpenAI-compatible endpoint across 240+ models and honors client routing directives, so you can swap models without rebuilding integrations or vendor SDKs.

3. Compress conversation history with summarization

Agents that retain full transcripts hit context limits and inflate input tokens quadratically. Summarize older turns into a rolling buffer every N exchanges to cap growth.

def compact(history, max_tokens=2000):
    if estimate_tokens(history) <= max_tokens:
        return history
    recent = history[-4:]
    old = history[:-4]
    summary = llm_summarize(old, model="gpt-4o-mini")
    return [{"role": "system", "content": f"Summary: {summary}"}] + recent

Run the summarizer on a cheap model. This keeps the agent grounded while preventing a 50-turn chat from becoming a 20k-token input on turn 51.

4. Truncate and shape tool outputs

Tool results—SQL rows, API JSON, file reads—are the biggest uncontrolled token source. Never pipe raw responses into the model. Define max length and pick fields.

def trim_tool_output(raw: dict, limit=500) -> str:
    lines = json.dumps(raw, indent=2).splitlines()
    if len(lines) > limit:
        return "\n".join(lines[:limit]) + f"\n... truncated {len(lines)-limit} lines"
    return "\n".join(lines)

Return only IDs and labels, not nested objects. If the tool supports pagination, request page size of 10. A single untrimmed Jira response can cost more than the rest of the turn combined.

5. Enforce strict structured outputs

Free-form text completion wastes tokens on filler and repeated disclaimers. Use JSON schema or function calling with strict mode to force minimal responses.

{
  "type": "json_schema",
  "schema": {
    "type": "object",
    "properties": {
      "action": {"type": "string", "enum": ["search", "reply", "escalate"]},
      "query": {"type": "string"}
    },
    "required": ["action"]
  }
}

Parsing becomes deterministic and you avoid “Sure, here is the answer” prefixes. Validate with pydantic or zod downstream; the model no longer decides how verbose to be.

6. Set max_tokens and stop sequences deliberately

Default max_tokens is often too high, letting the model ramble. Set it to the minimum viable for the task. Add stop tokens for repeated patterns.

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=msgs,
    max_tokens=128,
    stop=["\nUser:", "###"]
)

For extraction tasks, 64 tokens is frequently enough. Measure completion lengths in staging and tune. A misconfigured 2048 max_tokens on a yes/no classifier is pure waste.

7. Deduplicate embeddings and batch independent calls

If your agent embeds documents per request, cache embeddings by content hash. Batch independent LLM calls with asyncio.gather instead of sequential awaits.

import hashlib, asyncio
cache = {}
def embed(text):
    h = hashlib.md5(text.encode()).hexdigest()
    if h not in cache:
        cache[h] = client.embeddings.create(input=text, model="text-embedding-3-small").data[0].embedding
    return cache[h]

Batching cuts overhead tokens from repeated system prompts and reduces latency. Independent classification calls should never wait on each other.

8. Meter per-step usage and set guardrails

You cannot reduce token spend agents hide without measurement. Capture usage from each response and alert on outliers.

usage = resp.usage
if usage.total_tokens > BUDGET_PER_STEP:
    raise BudgetExceeded(usage)

n4n.ai provides per-token metering and automatic fallback when a provider is degraded; wire those events into your dashboards. When a primary model rate-limits, fall back to a cheaper one rather than retrying blindly.

Synthesis

Apply caching and truncation first; they are quick wins. Then route models and compact history before adding guardrails.