OpenTelemetry for LLM agents: a practical guide

Observability for autonomous systems demands more than standard APM dashboards. OpenTelemetry LLM agents requires treating model inference, tool execution, and agent loops as first-class spans with rich, queryable attributes. This guide lays out an ordered path to instrument a Python agent stack without drowning in telemetry noise.

1. Start with one span per agent invocation

Create a root span the moment an agent run begins. Everything else nests under it, giving you a single trace ID to correlate logs, metrics, and errors.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, OTLPSpanExporter

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter())
)

tracer = trace.get_tracer("agent.runtime")

def run_agent(task: str):
    with tracer.start_as_current_span("agent.run") as span:
        span.set_attribute("agent.task", task[:200])
        span.set_attribute("agent.version", "1.3.0")
        # ... agent logic

Keep the root span lean. Dump the full prompt into a span event or a log, not a high-cardinality attribute.

2. Emit a child span for every LLM request

The core challenge with OpenTelemetry LLM agents is that a single run may trigger dozens of model calls. Wrap each call explicitly so you can see model, token counts, and latency.

def call_llm(messages, model="gpt-4o"):
    with tracer.start_as_current_span("llm.call") as span:
        span.set_attribute("llm.model", model)
        span.set_attribute("llm.messages", len(messages))
        resp = client.chat.completions.create(model=model, messages=messages)
        span.set_attribute("llm.prompt_tokens", resp.usage.prompt_tokens)
        span.set_attribute("llm.completion_tokens", resp.usage.completion_tokens)
        span.set_attribute("llm.finish_reason", resp.choices[0].finish_reason)
        return resp

If you use an OpenAI-compatible gateway, capture the x-request-id header as an attribute. When scaling OpenTelemetry LLM agents to multi-agent systems, this per-call granularity is what lets you spot a single degraded model behind a vague “agent hung” symptom.

3. Represent tools and sub-agents as linked spans

Tool invocations are not log lines. They have latency, inputs, and failure modes. Create child spans for each tool call, and use span links when the agent fires parallel calls so the trace shows true concurrency.

def use_tool(name, args):
    with tracer.start_as_current_span(f"tool.{name}") as span:
        span.set_attribute("tool.args", json.dumps(args)[:500])
        try:
            result = dispatch(name, args)
            span.set_attribute("tool.ok", True)
        except Exception as e:
            span.record_exception(e)
            span.set_status(trace.Status(trace.StatusCode.ERROR))
            raise

For sub-agents, start a new span and pass the parent context explicitly if you cross process boundaries via RPC.

4. Record token spend as metrics, not just spans

Span attributes are great for debugging one call, but cost analysis needs aggregated metrics. Define OTel counters keyed by model.

from opentelemetry import metrics
meter = metrics.get_meter("agent.runtime")
prompt_counter = meter.create_counter("llm.prompt_tokens")
completion_counter = meter.create_counter("llm.completion_tokens")

# inside call_llm after response:
prompt_counter.add(resp.usage.prompt_tokens, {"model": model})
completion_counter.add(resp.usage.completion_tokens, {"model": model})

Tradeoff: avoid tagging metrics with user ID or task string. High-cardinality metric labels will blow up your backend. Aggregate by model and agent name only.

5. Propagate context across async and threaded boundaries

Agents frequently use asyncio or worker threads for tools. OpenTelemetry uses contextvars, but only if you propagate correctly.

import asyncio
from opentelemetry.context import Context, attach, detach

async def run_tool_async(ctx: Context, name, args):
    token = attach(ctx)
    try:
        with tracer.start_as_current_span(f"tool.{name}"):
            await dispatch_async(name, args)
    finally:
        detach(token)

If you spawn a thread, pass trace.get_current_span().get_span_context() and re-attach inside the worker. Missing this step silently orphans spans and breaks trace trees.

6. Forward gateway and cache signals into spans

If you route through a gateway such as n4n.ai, honor its cache-control hints and record them. The gateway may return x-cache: HIT or perform automatic fallback when a provider is degraded. Capture that as an attribute so you can correlate provider-side caching with latency spikes.

span.set_attribute("gateway.cache_hit", resp.headers.get("x-cache") == "HIT")
span.set_attribute("gateway.fallback_used", resp.headers.get("x-fallback") == "true")

This turns opaque gateway behavior into observable signal instead of a black box.

7. Export and query for agent-specific failure modes

Use OTLP export to a collector and write queries that target agent loops. Examples:

• Spans where llm.finish_reason != "stop" and agent.run duration > 30s.
• tool.ok == false grouped by tool.name.
• Traces with more than 20 llm.call spans (runaway loop).

Do not rely on default 1% sampling for production agents. Rare failure loops will vanish. Use tail-based sampling on span count or error status.

8. Common pitfalls and tradeoffs

Span explosion. Streaming token chunks as individual spans will overload your collector. Emit one span per request and log token deltas as events if needed.

PII leakage. Prompts contain user data. Never set full prompt text as an attribute. Truncate or hash.

Missing causality. Recording a tool error only in application logs loses the link to the parent agent run. Always use record_exception on the span.

Metric cardinality. Tagging token counters by trace_id creates millions of series. Keep labels coarse.

Sync vs async context. The most common bug we see: spans created inside a ThreadPoolExecutor have no parent because context wasn’t attached. Test your trace tree before shipping.

OpenTelemetry LLM agents is not just “add a tracer”. It is a deliberate data model: runs as roots, calls as children, tools as explicit spans, and token flow as metrics. Get that structure right and debugging a three-step agent becomes a two-minute trace query instead of a forensic log dive.