How to build an observability stack for LLM agents

Most LLM agent failures are silent until a user complains. Building an observability stack for LLM agents means capturing every model call, tool invocation, and retry so you can debug latency and cost spikes instead of guessing.

Step 1: Instrument the agent loop with OpenTelemetry

An agent is a loop: think, call a tool or model, observe, repeat. Wrap each iteration in a span so you get a timeline per run. OpenTelemetry (OTel) is the pragmatic default—vendor-neutral and supported by every trace backend.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer("agent")

span_exporter = OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(span_exporter))

Now wrap your step function. Keep spans coarse enough to be readable but fine enough to localize faults.

def run_step(prompt, step_num):
    with tracer.start_as_current_span(f"agent.step.{step_num}") as span:
        span.set_attribute("llm.prompt_chars", len(prompt))
        resp = call_model(prompt)
        span.set_attribute("llm.total_tokens", resp["usage"]["total_tokens"])
        span.set_attribute("llm.latency_ms", resp["latency_ms"])
        return resp

If your agent uses a ReAct pattern, emit a child span for each tool call inside the step span. That hierarchy is what makes a trace legible.

Step 2: Capture token usage and latency from provider responses

Every OpenAI-compatible endpoint returns a usage object. Parse it uniformly and attach it to the span. If you route through a gateway like n4n.ai, per-token usage metering is returned uniformly across 240+ models, and provider cache-control hints are forwarded, so your extraction logic doesn’t need per-vendor branches.

def call_model(prompt):
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    return {
        "content": resp.choices[0].message.content,
        "usage": {
            "prompt_tokens": resp.usage.prompt_tokens,
            "completion_tokens": resp.usage.completion_tokens,
            "total_tokens": resp.usage.total_tokens,
        },
        "latency_ms": resp.response_ms,
        "model": resp.model,
    }

Record latency as a numeric attribute, not a string. You’ll want histograms later.

Verify success

Run one prompt through run_step and confirm a span appears in your collector with llm.total_tokens set and llm.latency_ms > 0.

Step 3: Ship traces to a collector and backend

You need an OTLP receiver and a UI. Stand up the OTel collector and Jaeger with a minimal compose file.

services:
  otel-collector:
    image: otel/opentelemetry-collector:0.102.0
    command: ["--config=/etc/otel.yaml"]
    volumes:
      - ./otel.yaml:/etc/otel.yaml
    ports:
      - "4318:4318"
  jaeger:
    image: jaegertracing/all-in-one:1.57
    ports:
      - "16686:16686"

otel.yaml:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [jaeger]

Run docker compose up -d. Open http://localhost:16686 and search for the agent service. If you see your span, the pipeline works.

Step 4: Correlate logs with trace IDs

Agents log tool calls, retries, and parsing errors. Without trace correlation those logs are noise. Inject the current span context into every log line.

import logging
from opentelemetry.trace import get_current_span

logging.basicConfig(format="%(message)s")
logger = logging.getLogger(__name__)

def tool_call(name, args):
    span = get_current_span()
    ctx = span.get_span_context()
    logger.info(f"trace_id={ctx.trace_id:x} span_id={ctx.span_id:x} tool={name} args={args}")
    # invoke the actual tool here

Ship these logs to Loki or Elasticsearch. Filter by trace_id to reconstruct a single agent run from both traces and logs.

Step 5: Track provider routing and fallback

Production agents rarely call one model. They route by task, fall back on 429s, or use a gateway. Record which provider actually served the request. A gateway with automatic fallback when a provider is rate-limited or degraded reduces blind spots; ensure your observability stack records which provider actually served the request.

span.set_attribute("llm.provider", resp.get("provider", "unknown"))
span.set_attribute("llm.model", resp.get("model"))
if resp.get("fallback_used"):
    span.set_attribute("llm.fallback", True)

If you send client routing directives (e.g., “prefer anthropic”), log the directive and the effective route. That exposes mismatches between intent and reality.

Step 6: Build dashboards for cost and latency

Traces are great for debugging, but trends need metrics. Use the OTel collector’s Prometheus exporter or scrape a metrics sidecar.

Key signals:

agent_step_latency_ms (histogram by model)
agent_tokens_total (counter by model and provider)
agent_error_rate (span status error / total)

Grafana panel (PromQL):

sum(rate(agent_tokens_total[5m])) by (model)

Alert rule: rate(agent_error_rate[10m]) > 0.05. That catches a degraded tool or a silently failing parser before users churn.

Step 7: End-to-end verification with a synthetic run

Automate a smoke test that asserts the stack works. A pytest fixture is enough.

def test_observability_emits_span():
    resp = run_step("What is 2+2? Return only the number.")
    assert resp["usage"]["total_tokens"] > 0
    # hit Jaeger HTTP API to confirm span persisted
    import requests
    r = requests.get("http://localhost:16686/api/traces?service=agent")
    assert r.status_code == 200
    assert len(r.json()["data"]) >= 1

Run it in CI after every agent change. If spans stop flowing, the test fails and you know before prod does.

Step 8: Handle cache hits, retries, and partial failures

Agents retry on timeout. Mark retries so latency isn’t masked.

for attempt in range(3):
    with tracer.start_as_current_span("agent.retry") as span:
        span.set_attribute("retry_count", attempt)
        try:
            return call_model(prompt)
        except TimeoutError:
            span.set_status(trace.Status(trace.StatusCode.ERROR))

If the provider returns a cached completion, the forwarded cache-control hint tells you. Log it.

if resp.get("cache_hit"):
    span.set_attribute("llm.cache_hit", True)

This surfaces false economies: a high cache hit rate may mean you’re reusing stale contexts.

What a mature observability stack for LLM agents looks like

You now have traces per run, logs correlated by trace ID, metrics for cost and latency, and alerts on error rate. The observability stack for LLM agents you built is portable: swap Jaeger for Tempo, or HTTP OTLP for gRPC, without touching instrumentation. The hard part—making agent behavior visible instead of guessed—is done.