Most agent observability stacks stop at span-level token counts. Token-level tracing AI agents goes deeper: it records which exact tokens triggered a tool call, which were served from cache, and how prompt prefixes shifted across iterations. That granularity turns black-box agent loops into auditable decision streams.
Why span-level metrics hide agent failures
A span that reports prompt_tokens: 4200, completion_tokens: 180 tells you the call was expensive. It does not tell you that 3,900 of those prompt tokens were redundant repetitions of the system prompt because your agent loop mutated the message array incorrectly. It does not tell you the model emitted a malformed tool call that your parser silently dropped, wasting 40 completion tokens.
When you debug agents in production, the bug is rarely “the model was wrong.” It is “the prompt we built was not the prompt we thought we built.” Token-level tracing AI agents exposes the prompt construction and generation process as data, not guesswork.
Step 1: Capture raw token events at the boundary
Intercept every model call at the HTTP boundary. If you route through a gateway, you get one place to hook. A gateway like n4n.ai that exposes an OpenAI-compatible endpoint across 240+ models and meters per-token usage gives you a single interception point for token-level tracing AI agents without per-provider glue.
Wrap the client so you always stream and ask for usage:
from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="sk-...")
def traced_chat(messages, model="anthropic/claude-3.5-sonnet", **kwargs):
chunks = []
usage_total = None
stream = client.chat.completions.create(
model=model,
messages=messages,
stream=True,
stream_options={"include_usage": True},
)
for chunk in stream:
if chunk.usage:
usage_total = chunk.usage
if chunk.choices and chunk.choices[0].delta.content:
chunks.append(chunk.choices[0].delta.content)
return "".join(chunks), usage_totalStreaming adds marginal latency but gives you chunk boundaries. If you cannot stream, fall back to non-streaming usage in the response—you lose intra-call timing but keep totals.
Step 2: Attribute tokens to agent phases
An agent run is a sequence of phases: retrieve, plan, act, observe, reflect. Tag each message with a phase identifier before sending. Store the tag alongside the token count for that call.
messages = [
{"role": "system", "content": sys_prompt, "meta": {"phase": "init"}},
{"role": "user", "content": query, "meta": {"phase": "retrieve"}},
]After the call, record:
trace_log.append({
"phase": "retrieve",
"model": model,
"prompt_tokens": usage_total.prompt_tokens,
"completion_tokens": usage_total.completion_tokens,
"call_id": call_id,
})Do not rely on span names from your framework. Framework spans often group multiple model calls under one “agent step.” You need the per-call phase mapping to see where tokens actually went.
Step 3: Decode cached vs generated tokens
Providers that support prompt caching return breakdown details. OpenAI-compatible responses include prompt_tokens_details when caching is active:
{
"prompt_tokens": 1820,
"completion_tokens": 64,
"prompt_tokens_details": {
"cached_tokens": 1500
}
}If cached_tokens is low on calls where you expected a stable prefix, your message array is shifting. A common cause: injecting a timestamp or request ID into the system prompt on every iteration. That breaks the cache prefix and silently multiplies cost. Token-level tracing AI agents makes this visible because you watch cached_tokens drop to zero across loops.
Step 4: Reconstruct tool-call boundaries
Function-calling agents emit structured tokens for function_call or tool blocks. Capture the substring where the tool name first appears and the closing brace. This lets you measure how many completion tokens were spent formatting the call versus reasoning beforehand.
import re
def split_tool_call(text):
m = re.search(r"<tool_call>(.*?)</tool_call>", text, re.DOTALL)
if m:
return text[:m.start()], m.group(1)
return text, NoneIf the pre-tool reasoning span grows across iterations, your agent is looping. That pattern shows up clearly only when you trace tokens per call, not when you look at daily aggregate spend.
Step 5: Store traces for post-hoc replay
Write traces to a columnar store or even JSONL. Keep the raw prompt token estimate (using the provider’s tokenizer or tiktoken for OpenAI models) alongside the provider’s count. Mismatch is expected for non-OpenAI models; record both.
{"run_id":"r1","call_id":"c3","phase":"act","model":"anthropic/claude-3.5-sonnet","prompt_tokens":1820,"cached_tokens":1500,"completion_tokens":64,"tool":"search"}Replay scripts can then reconstruct the exact message list for any call id if you also snapshot the message array (minus raw document contents if privacy-sensitive).
Common pitfalls and tradeoffs
Tokenizer drift. tiktoken cl100k is wrong for Llama or Claude. Use the provider’s returned counts as source of truth. Local tokenizers are only for estimating before the call.
Storage cost. Full token-level tracing AI agents means storing per-call metadata for every run. At 10k runs/day with 5 calls each, that is 50k small records—manageable in Postgres, but decide retention up front. Raw prompt snapshots are the expensive part; hash them and store blobs in object storage.
Streaming overhead. Some legacy load balancers buffer streams. Verify your gateway forwards chunks uninterrupted, or your include_usage event arrives only at the end, reducing timeliness.
Privacy leakage. Prompt snapshots contain user data. Redact before writing to long-term trace stores. Cache hit rates can be computed without storing the cached prefix.
False precision. Token counts are not thought counts. A low completion token count does not mean a simple decision. Use tracing to locate anomalies, then read the actual text.
Production checklist
- All model calls routed through one instrumented boundary.
-
stream_options.include_usageenabled on every streaming call. - Phase tags attached to messages before send.
-
prompt_tokens_details.cached_tokensalerted when below expected threshold. - Tool-call token splits computed for at least the top three tools.
- Trace retention policy set; raw prompts redacted or externed.
- Replay script validated against last 100 production runs.
Token-level tracing AI agents is not about watching tokens for their own sake. It is about making the agent’s internal contract with the model explicit: what we sent, what we reused, what we wasted. Ship the capture first, build dashboards later. The first time a cache miss triples your bill, the trace will pay for itself.