How to debug a failing AI agent with trace logs

A multi-step agent that worked yesterday now returns malformed JSON, and the only clue is a generic 500 from the orchestrator. To debug AI agent trace logs effectively, you need structured span data that captures each LLM call, tool invocation, and retry under one correlation ID—not a pile of unstructured print() statements. Application logs tell you what happened in one process; traces tell you why the chain broke.

Step 1: Instrument your agent with OpenTelemetry spans

Wrap every logical unit of work in an OpenTelemetry span. At minimum, create a root span for the agent run, a child span for each planner or reasoning LLM call, a child for each tool execution, and a child for final synthesis. If you use async Python, propagate the context correctly or your spans will flatten into unrelated rows.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("agent.debug")

async def run_agent(query: str):
    with tracer.start_as_current_span("agent_run") as root:
        root.set_attribute("input.query", query)
        plan = await call_llm(root, "plan", PLANNER_PROMPT, query)
        for i, step in enumerate(plan["steps"]):
            with tracer.start_as_current_span(f"step.{i}") as step_span:
                step_span.set_attribute("tool.name", step["tool"])
                await execute_tool(step_span, step)

The parent-child relationship is what lets you debug AI agent trace logs by expanding the tree instead of grepping for a request ID across files. If you skip child spans and just log inside the loop, you lose the ordering when two steps run concurrently.

Step 2: Capture LLM request and response payloads without leaking secrets

A span that only says llm.call took 800ms is useless. Attach the model name, token usage, and a truncated copy of the completion. Redact any user PII before export—do not ship raw prompts to a shared tracing backend.

async def call_llm(parent_span, role: str, system_prompt: str, user_msg: str = ""):
    with tracer.start_as_current_span(f"llm.{role}") as llm_span:
        resp = await openai.ChatCompletion.acreate(
            model="gpt-4o-mini",
            messages=[{"role": "system", "content": system_prompt},
                      {"role": "user", "content": user_msg}],
        )
        llm_span.set_attribute("llm.model", resp.model)
        llm_span.set_attribute("llm.prompt_tokens", resp.usage.prompt_tokens)
        llm_span.set_attribute("llm.completion_tokens", resp.usage.completion_tokens)
        llm_span.set_attribute("llm.output", resp.choices[0].message.content[:500])
        return resp

If you route through n4n.ai, the gateway automatically falls back when a provider is rate-limited and forwards cache-control hints, so your span should record the x-provider response header to know which backend actually served the token. Add that header to the span attributes:

llm_span.set_attribute("llm.provider", resp.headers.get("x-provider", "unknown"))
llm_span.set_attribute("llm.cached", resp.headers.get("x-cache", "MISS") == "HIT")

When you later debug AI agent trace logs from a run that mixed providers, you can see whether a slow step was a model problem or a fallback event.

Step 3: Correlate tool calls with model decisions

The most frequent agent failure is a tool returning an unexpected shape, causing the next LLM call to invent a workaround. Log the exact tool input and output as span events, not just attributes, so you preserve ordering within the span.

async def execute_tool(span, step: dict):
    span.add_event("tool.request", {"input": json.dumps(step["args"])})
    try:
        result = await dispatch(step["tool"], step["args"])
        span.add_event("tool.response", {"output": str(result)[:500]})
    except Exception as e:
        span.record_exception(e)
        span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
        raise

A planner span might show the model asked for order_id: "123". The tool span shows tool.response: "null". The following synthesize span then errors because it tried to index null["status"]. That chain is only visible if the three spans share a parent and carry the payloads. To debug AI agent trace logs at this level, you must resist the urge to catch and swallow tool errors—let the span record them.

Step 4: Reconstruct the failure path from trace logs

Export spans to Jaeger, Tempo, or any backend that supports trace queries. When a user reports a failure, grab the trace ID from your request log and open the waterfall view. Look for the first span with status.code = ERROR or an exception event.

A minimal broken trace looks like this:

{
  "traceID": "a1b2c3",
  "spans": [
    {"name": "agent_run", "attributes": {"input.query": "refund status"}},
    {"name": "llm.plan", "attributes": {"llm.model": "gpt-4o-mini", "llm.prompt_tokens": 420}},
    {"name": "step.0", "events": [
       {"name": "tool.request", "attributes": {"input": "{\"order_id\": \"123\"}"}},
       {"name": "tool.response", "attributes": {"output": "null"}}
    ]},
    {"name": "llm.synthesize", "status": {"code": "ERROR", "message": "TypeError"}}
  ]
}

The tool.response of null is the root cause. The synthesize step failed because the planner assumed a dict. Fix the tool contract, not the prompt. If you debug AI agent trace logs without the event payloads, you will waste an hour rewriting the system message.

For high-volume systems, write a quick LogQL or TraceQL query to surface all agent runs where a tool.response contained "null" and the root span errored:

{app="agent"} | json | span_name="tool_exec" | json | output=`null` 

Step 5: Add routing and fallback context to spans

Agents that call multiple providers need explicit routing attributes. A span attribute llm.fallback set to true tells you the call was retried on another backend after a rate limit. This separates genuine model mistakes from infrastructure hiccups.

if resp.headers.get("x-fallback") == "true":
    llm_span.set_attribute("llm.fallback", True)
    llm_span.set_attribute("llm.original_provider", resp.headers.get("x-original-provider"))

Also record client routing directives if you send them. Some gateways honor a header like x-router: prefer-anthropic and will forward cache-control hints. Capture that hint on the span so you can prove the cache was used:

llm_span.set_attribute("llm.cache_control", resp.headers.get("x-cache-control", "none"))

When you debug AI agent trace logs across a fleet, these fields let you filter “all errors on fallback calls” versus “all errors on primary calls” in seconds.

Step 6: Verify your debugging setup with a forced failure

Do not wait for production to break. Inject a fault in a staging run: make the tool return None or raise a timeout. Run the agent, then confirm your trace backend shows the breakdown.

def faulty_tool(args):
    raise TimeoutError("db unreachable")

# in test harness, swap dispatch for faulty_tool

After the run, open the trace and verify three things:

1. The step.0 span is marked ERROR with the TimeoutError recorded.
2. The parent agent_run span shows the full child hierarchy and total duration.
3. The LLM plan span before the failure contains the full prompt and token counts.

If those hold, your instrumentation is sufficient. You can now debug AI agent trace logs from any future incident without adding temporary logging.

Step 7: Keep spans lean and searchable

Once this is in production, set a max attribute length and drop verbose stack traces from spans (keep them in error events). High-cardinality attributes like input.query are fine; random UUIDs as attribute values are not. Sample at 100% in staging, and use tail-based sampling in prod to keep all error traces.

from opentelemetry.sdk.trace.export import BatchSpanProcessor
# use BatchSpanProcessor in prod, not SimpleSpanProcessor
provider.add_span_processor(BatchSpanProcessor(OTLPE exporter))

A trace that is too noisy is as bad as no trace. The goal is to debug AI agent trace logs by finding the one broken step in a 12-step run, not scrolling through 400 events of token streams.

Verify success

Success means a new agent failure ticket gets resolved by opening one trace link. If an engineer needs to SSH into a pod and read local logs, the tracing setup is incomplete. Measure this by running a game-day: break a tool, page the on-call, and time how long from alert to root-cause commit. With proper spans, that should drop from hours to minutes.