Agent loops waste tokens and latency when they call the same tool with identical arguments across turns. Caching tool results agents issue lets you skip redundant computations and model round-trips by memoizing deterministic outputs at the orchestration layer. This article walks through a concrete implementation you can drop into a production Python service.
Step 1: Identify deterministic tools and define cache keys
Not every tool is safe to cache. Pure functions, read-only API GETs, and idempotent lookups are candidates. Anything that mutates state, depends on hidden time, or returns non-deterministic data must stay uncached.
Build a cache key from the tool name and a canonical serialization of its arguments. Sort JSON keys to avoid mismatch from dict ordering.
import hashlib
import json
def make_cache_key(tool_name: str, args: dict) -> str:
canonical = json.dumps(args, sort_keys=True, separators=(",", ":"))
raw = f"{tool_name}:{canonical}"
return hashlib.sha256(raw.encode()).hexdigest()For caching tool results agents correctly, exclude volatile fields like request_id or timestamp from the args before hashing. If a tool accepts a force_refresh flag, branch around the cache.
Step 2: Choose a cache backend with TTL and eviction
In-process dict works for single-process prototypes but loses state on restart and ignores cross-worker sharing. Use Redis in any distributed deployment.
import redis
r = redis.Redis(host="localhost", port=6379, db=0, decode_responses=True)
def cache_get(key: str) -> str | None:
return r.get(key)
def cache_set(key: str, value: str, ttl_seconds: int) -> None:
r.setex(key, ttl_seconds, value)Set TTL based on data freshness requirements. Stock prices: 30 seconds. Documentation lookups: 1 hour. Never cache forever unless the source is immutable.
Step 3: Wrap tool execution with cache lookup and store
Write a decorator that intercepts calls. Keep the original tool callable untouched for non-cacheable paths.
from functools import wraps
from typing import Callable
def cached_tool(ttl: int = 300):
def decorator(fn: Callable):
@wraps(fn)
def wrapper(tool_input: dict):
key = make_cache_key(fn.__name__, tool_input)
hit = cache_get(key)
if hit is not None:
return json.loads(hit)
result = fn(tool_input)
cache_set(key, json.dumps(result), ttl)
return result
return wrapper
return decoratorApply it to a concrete tool:
@cached_tool(ttl=600)
def fetch_github_repo(tool_input: dict) -> dict:
# real HTTP call omitted
return {"stars": 1234, "lang": "Python"}This pattern makes caching tool results agents invisible to the agent prompt logic—the tool just gets faster.
Step 4: Integrate with the agent loop
Assume a minimal loop that calls an LLM, gets a tool call, executes it, and feeds the result back. Patch the execution step only.
def run_agent_loop(messages, tools, max_steps=10):
for _ in range(max_steps):
resp = llm_chat(messages, tools=tools)
if not resp.tool_calls:
return resp.content
for call in resp.tool_calls:
fn = TOOL_REGISTRY[call.name]
# fn is already wrapped with @cached_tool
result = fn(call.arguments)
messages.append({"role": "tool", "content": json.dumps(result)})
return "loop exhausted"If you use LangChain, wrap the Tool object’s _run method or use a BaseTool subclass that checks cache before calling the parent. The key is that the agent never sees the cache; it only benefits from lower latency.
Step 5: Handle invalidation and versioning
TTL covers time-based staleness, but schema changes break deserialization. Prefix keys with a version string.
CACHE_VERSION = "v2"
def make_cache_key(tool_name: str, args: dict) -> str:
canonical = json.dumps(args, sort_keys=True, separators=(",", ":"))
raw = f"{CACHE_VERSION}:{tool_name}:{canonical}"
return hashlib.sha256(raw.encode()).hexdigest()Bump CACHE_VERSION on any tool output shape change. For manual invalidation (e.g., admin action), expose a purge_tool(tool_name) that scans Redis keys with a prefix—or maintain a set of known keys per tool to avoid KEYS scans in production.
Step 6: Forward cache-control hints to model providers where relevant
Tool-result caching lives in your orchestration code, but prompt caching at the LLM layer is complementary. Some inference gateways, including n4n.ai, forward provider cache-control hints to upstream models so repeated system prompts are cheaper. Use both: cache tool outputs locally, and let the gateway cache static prompt prefixes.
Do not confuse the two. A cached tool result still gets sent to the model every turn unless you also trim conversation history. Keep tool responses concise to limit token bleed.
Step 7: Verify success with hit-rate and latency metrics
Instrument the wrapper. Count hits and misses, and log elapsed time for cache misses versus total calls.
import time
from collections import defaultdict
stats = defaultdict(int)
def cached_tool(ttl: int = 300):
def decorator(fn: Callable):
@wraps(fn)
def wrapper(tool_input: dict):
key = make_cache_key(fn.__name__, tool_input)
hit = cache_get(key)
if hit is not None:
stats["hits"] += 1
return json.loads(hit)
stats["misses"] += 1
start = time.monotonic()
result = fn(tool_input)
stats["miss_latency"] += time.monotonic() - start
cache_set(key, json.dumps(result), ttl)
return result
return wrapper
return decoratorEmit these stats to your metrics system (Prometheus, Datadog). A healthy loop shows hit rate climbing above 40% after warmup on repetitive tasks like codebase Q&A or customer ticket triage.
Verify end to end: run the agent on a fixed scenario twice. The second run should show zero outbound calls to cached tools (mock the HTTP layer or watch Redis GET counts). Measure per-loop token usage from your provider’s usage field; it should drop because tool outputs are served from memory instead of recomputed, and the model still receives them but you avoided the upstream API cost.
Edge cases and opinions
Cache only tools that are genuinely pure. I have seen teams cache a “send_email” tool behind a flag and then wonder why duplicates vanished. Separate side effects from queries.
Use short TTLs by default. A 5-minute cache on a rapidly changing API is safer than a 1-day cache that silently serves stale data. You can always raise TTL after observing hit rates and confirming data stability.
For multi-tenant agents, scope the cache key with tenant_id. Never let one customer’s cached result leak to another.
If your agent calls the same tool with the same args in parallel turns, add a lock or use Redis SET NX to prevent thundering herd misses. A simple redis.lock.Lock around the miss path works.
Closing implementation checklist
- List tools; mark deterministic ones.
- Implement
make_cache_keywith version prefix. - Stand up Redis; set per-tool TTL constants.
- Wrap tools with
cached_tool. - Inject wrapped tools into agent registry.
- Add hit/miss metrics and alert on low hit rate.
- Test replay scenario to confirm cache serves and no extra upstream calls fire.
Follow these steps and caching tool results agents becomes a default optimization rather than an afterthought. Your loops get faster, your bill drops, and the model stays none the wiser.