Most multi-agent prototypes collapse under their own context growth. Effective token budgeting multi-agent systems requires treating tokens as a scarce, metered resource from day one, not as an afterthought when the bill arrives.
Why unbounded agents burn through context
In a typical multi-agent setup, a coordinator spawns worker agents and forwards conversation history so each worker has “full context.” That feels productive until the third iteration, when the combined prompt size exceeds the model’s context window or the cost per task triples. Token budgeting multi-agent designs forces you to decide what each agent actually needs to see.
The root cause is treating context as free. It isn’t. Every forwarded message is tokens in, and often tokens out. Without a hard ceiling, a single stuck task can loop, respawn workers, and drain your quota in minutes. The explosion is multiplicative: N agents each receiving M messages of growing size.
Step 1: Define a global token ceiling per run
Start by setting a maximum token spend for a single agentic task episode. This is not the model’s context window; it’s your economic boundary. Store it in configuration and pass it to the orchestrator.
import os
class BudgetExceeded(Exception):
pass
class TokenBudget:
def __init__(self, max_tokens: int = 100_000):
self.max_tokens = max_tokens
self.used = 0
def consume(self, prompt_tokens: int, completion_tokens: int):
spent = prompt_tokens + completion_tokens
if self.used + spent > self.max_tokens:
raise BudgetExceeded(f"Run budget {self.max_tokens} exceeded")
self.used += spent
def remaining(self) -> int:
return self.max_tokens - self.usedPick a number from unit economics
Set the ceiling based on the task value, not the model limit. A customer-facing free-tier task should have a far lower cap than an internal data-processing job. If a task historically consumes 20k tokens and you want margin, set 30k. Don’t default to 200k because the model supports it.
Separate context window from spend ceiling
The context window is a technical limit per call. Your budget is cross-call and cross-agent. A 128k context model can still blow a 50k task budget in two calls if completions are long.
Step 2: Allocate per-agent sub-budgets
Break the global ceiling into role-based allowances. A planner might get 30%, a coder 40%, a critic 20%, and a summarizer 10%. Encode this in static config so you can tune without code changes.
{
"agents": {
"planner": { "weight": 0.3, "max_context": 8000 },
"coder": { "weight": 0.4, "max_context": 16000 },
"critic": { "weight": 0.2, "max_context": 6000 },
"summarizer":{ "weight": 0.1, "max_context": 4000 }
}
}Weighted vs hard caps
The weight drives dynamic reallocation if one agent under-spends. The max_context is a hard cap on prompt size for that agent. Enforce max_context before sending. If a message pack exceeds it, truncate or summarize.
Dynamic reallocation
If the critic finishes early, fold its remaining weight into the coder. Implement a simple ledger:
def reallocate(budget, agent, unused):
budget.used -= unused # credit back
# hand to next agent by raising its session capAvoid micro-managing every call; adjust at agent boundaries.
Step 3: Truncate and summarize at the edge
You need a deterministic way to shrink context. Two patterns work: sliding window (keep last N tokens) and hierarchical summary (compress older turns). Sliding window is cheap but loses long-term goals. Summary adds a token cost but preserves intent.
Sliding window implementation
Use a tokenizer to measure before sending:
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
def truncate_messages(messages, max_tokens: int):
out = []
total = 0
for msg in reversed(messages):
t = len(enc.encode(msg["content"]))
if total + t > max_tokens:
break
out.insert(0, msg)
total += t
return outHierarchical summarization tradeoffs
For multi-agent flows, route stale context to a summarizer agent only when the window would drop critical instructions. That tradeoff is where most teams waste budget—summarizing too eagerly spends tokens; too late loses coherence. Cap the summarizer’s output at 10% of the truncated text.
Cache static prefixes
System prompts and tool schemas rarely change within a run. Mark them with provider cache-control so repeated sends hit cache. This is invisible token savings.
curl https://your-gateway/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o-mini",
"messages": [{"role": "system", "content": "static spec"}],
"cache_control": {"type": "ephemeral"}
}'Step 4: Meter every call and deduct immediately
Do not estimate post-hoc. Use the usage field from the completion response to deduct exact counts. If you route through a gateway such as n4n.ai, per-token usage metering is returned on each response, letting you deduct from the budget without re-implementing counters.
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=truncated,
)
budget.consume(
resp.usage.prompt_tokens,
resp.usage.completion_tokens,
)Threshold actions
When remaining() drops below 5% of global, switch agents to a cheaper model or halt spawning. That fallback prevents a death spiral where the system spends its last tokens on a low-value critique.
Step 5: Route with fallback to protect the budget
Provider rate limits and degraded latencies cause naive retries that multiply token spend. Honor client routing directives and forward provider cache-control hints so repeated context hits cache instead of recomputing.
Retry storms
A 429 with a full prompt resend can double cost. Implement capped fallback:
try:
resp = call_model("primary", truncated)
except RateLimitError:
truncated = truncate_messages(truncated, max_tokens // 2)
resp = call_model("fallback-small", truncated)Automatic fallback when a provider is degraded is useful, but only if the fallback tier is cheaper. Otherwise you trade latency for budget violation.
Common pitfalls and tradeoffs
Over-truncation. Cutting context to meet a tight budget yields agents that repeat mistakes. Keep at least the original task spec and latest feedback.
Summary inflation. A summarizer that outputs 2x the truncated text defeats the purpose. Cap its output tokens explicitly with max_tokens.
Synchronous metering bottlenecks. If every agent call blocks on a central budget service, you serialize your fleet. Use an in-memory atomic counter per run; persist only at checkpoints.
Ignoring output tokens. Input budgeting is visible; completion tokens sneak up. Set max_tokens on every generation call.
Cache misses. Multi-agent systems often resend identical system prompts. Use provider cache-control to mark static prefixes; the gateway must forward those hints or you lose the benefit.
No per-agent isolation. A buggy worker can consume the entire budget. Enforce sub-budgets strictly; raise BudgetExceeded inside the agent loop.
Minimal reference orchestrator
Below is a compact loop that wires the pieces together. It is not production-complete but shows the control flow.
def run_task(task_desc: str, budget: TokenBudget):
messages = [{"role": "system", "content": task_desc}]
agents = load_agent_config()
while not task_done(messages):
for name, cfg in agents.items():
sub_max = cfg["max_context"]
window = truncate_messages(messages, sub_max)
try:
resp = call_agent(name, window)
except RateLimitError:
window = truncate_messages(window, sub_max // 2)
resp = call_agent(f"{name}-small", window)
budget.consume(resp.usage.prompt_tokens, resp.usage.completion_tokens)
messages.append({"role": "assistant", "content": resp.choices[0].message.content})
if budget.remaining() < budget.max_tokens * 0.05:
messages = summarize_history(messages, budget)
if budget.remaining() <= 0:
raise BudgetExceeded("halted")Token budgeting multi-agent systems is iterative engineering. Measure real runs, adjust weights, and keep the ceiling strict. The teams that ship reliable agentic products are the ones that treat tokens like the constrained operational resource they are.