n4nAI
Open console →
Developer toolsHow-toOpenTelemetry Tracing for LLM Apps

Debugging slow LLM chains using OpenTelemetry waterfalls

Learn how to trace and debug LLM chains with OpenTelemetry waterfalls. Step-by-step instrumentation, span analysis, and latency fixes for Python apps.

n4n Team4 min read836 words

Audio narration

Coming soon — every post will get a voice note here.

A slow LLM chain often fails silently—a single 2s retrieval call or a misconfigured retry can balloon end-to-end latency. debugging LLM chains with OpenTelemetry waterfalls gives you a per-span timeline that shows exactly where tokens and seconds go, instead of guessing from aggregated metrics.

Step 1: Initialize the OpenTelemetry tracer in your process

Before you can see a waterfall, you need a tracer instance and a span processor. In Python, the opentelemetry-sdk provides a TracerProvider that you configure once at startup. The example below uses a console exporter so you can confirm spans are created before wiring a real backend.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("retrieval_chain.v1")

If you run a multi-process service (e.g., FastAPI workers), set the provider in the entrypoint before any request handling. Spans created without a provider are no-ops, and you’ll stare at an empty waterfall wondering why. For production, set a sampler (ParentBasedTraceIdRatioBased) to avoid 100% capture on every request once your volume grows.

Step 2: Wrap each LLM call as a child span

The core of debugging LLM chains with OpenTelemetry waterfalls is granular spans around model inference, retrieval, and any post-processing. Don’t just wrap the whole chain in one span—break it down. Below is a manual wrapper for an OpenAI chat call that records model, token counts, and errors using the emerging gen_ai semantic conventions.

import openai
from opentelemetry.trace import Status, StatusCode

def chat_with_span(model: str, messages: list):
    with tracer.start_as_current_span("llm.chat") as span:
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.request.message_count", len(messages))
        span.set_attribute("gen_ai.request.temperature", 0.0)
        try:
            resp = openai.ChatCompletion.create(model=model, messages=messages, temperature=0.0)
            span.set_attribute("gen_ai.response.total_tokens", resp.usage.total_tokens)
            span.set_attribute("gen_ai.response.completion_tokens", resp.usage.completion_tokens)
            return resp
        except openai.error.RateLimitError as e:
            span.set_status(Status(StatusCode.ERROR, "rate_limited"))
            span.record_exception(e)
            raise

For LangChain or LlamaIndex, use the OpenInference instrumentation libraries (openinference-instrumentation-openai, openinference-instrumentation-langchain) which auto-create spans with the right attributes. That saves you from hand-rolling instrumentation and keeps your waterfall consistent across refactors. The key is that every external boundary—vector DB query, LLM call, validation step—gets its own span.

Step 3: Export OTLP traces to a waterfall backend

Console output is unreadable at scale. Point the BatchSpanProcessor at an OTLP collector (Jaeger, Tempo, or Grafana) so you get a real waterfall UI.

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(otlp_exporter))

Launch Jaeger all-in-one via Docker to follow along locally:

docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 -p 4317:4317 \
  jaegertracing/all-in-one:latest

Once your app makes a request, open http://localhost:16686, find the retrieval_chain.v1 service, and click a trace. You’ll see a waterfall: horizontal bars stacked by parent-child relationship, with x-axis as time. Each bar’s left edge is start, right edge is end; indentation shows nesting.

Step 4: Read the waterfall to locate latency thieves

A waterfall makes three failure modes obvious:

  1. Sequential bottlenecks. If your retrieve_docs span (300ms) and llm.chat span (1800ms) are stacked vertically, they ran serially. If retrieval is independent of the first LLM call, move them into parallel spans using asyncio.gather with separate start_as_current_span contexts.
  2. Idle gaps. White space between the end of a child span and start of the next means your code blocked on something untraced—often a sleep, a network retry, or a poorly tuned connection pool.
  3. Provider variance. When debugging LLM chains with OpenTelemetry waterfalls, tag each span with the resolved provider. A span that shows gen_ai.request.model = gpt-4 but took 12s while another call with same model took 2s indicates a degraded endpoint or queuing at the gateway.

Example span structure

{
  "name": "llm.chat",
  "start_time": "2024-05-01T10:00:00.000Z",
  "end_time": "2024-05-01T10:00:01.800Z",
  "attributes": {
    "gen_ai.request.model": "gpt-4",
    "gen_ai.response.total_tokens": 1200,
    "gen_ai.request.provider": "openai",
    "gen_ai.request.route": "fallback:anthropic"
  },
  "parent_span_id": "abc123"
}

Emit provider routing as attributes:

span.set_attribute("gen_ai.request.provider", "openai")
span.set_attribute("gen_ai.request.route", "fallback:anthropic")

If you front your models with a gateway such as n4n.ai, it honors client routing directives and forwards provider cache-control hints; emit those as span attributes to see cache hits in the waterfall. A cache_control: hit attribute on a child span explains why one call is 40ms and another 2s. Without that visibility, you’ll wrongly assume the model itself is slow.

Step 5: Add context propagation across async boundaries

Many chains use asyncio or thread pools. OpenTelemetry context must be explicitly carried or your waterfall fragments into disconnected traces. Use contextvars and the otel context API:

from opentelemetry.context import attach, detach, get_current

ctx = get_current()
token = attach(ctx)
# pass ctx into worker
try:
    await worker()
finally:
    detach(token)

In FastAPI, middleware like opentelemetry-instrumentation-fastapi does this automatically. Verify the trace ID is identical across spans in the UI; if each span is its own trace, you lost propagation at a await or ThreadPoolExecutor boundary. This is the most common reason engineers think tracing “doesn’t work” when in fact they dropped context.

Step 6: Verify success by comparing before/after waterfalls

The only real proof a fix worked is a shorter waterfall with the same logical steps. Capture a trace before optimization, note the total duration and span count. After changes—say, parallelizing retrieval or enabling provider caching—capture a new trace and confirm:

  • End-to-end span duration dropped.
  • No new error spans appeared (fallbacks should be explicit, not silent exceptions).
  • Token usage attributes match expectations (cached calls show lower completion_tokens or a cache_hit flag).

You can extract a quick p95 from span durations using the OTLP data in your backend, or run a small script:

from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult

class DurationAggregator(SpanExporter):
    def export(self, spans):
        for s in spans:
            if s.name == "llm.chat":
                print(s.end_time - s.start_time, "ns")
        return SpanExportResult.SUCCESS

Wire that as an additional processor temporarily to log raw nanosecond deltas. If the median llm.chat span moves from 1800ms to 400ms after you enabled prompt caching, your waterfall validates the change.

Beyond the steps: attributes that pay off

When debugging LLM chains with OpenTelemetry waterfalls, the timeline tells you when, but attributes tell you why. At minimum emit:

  • gen_ai.request.model
  • gen_ai.request.temperature
  • gen_ai.response.total_tokens
  • gen_ai.request.provider and route
  • error.type on exceptions

With those, a waterfall becomes a forensic tool: you can filter all traces where provider = slow-host and duration > 5s, then fix the routing rule. Stick to these steps and you’ll turn “the chain feels slow” into a specific PR that removes redundant serialization or a hidden retry loop. The waterfall doesn’t lie.

Tagsopentelemetrydebugginglatencytracing

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Developer toolsCross-service LLM span correlation with OpenTelemetryA practical guide to correlating LLM spans across microservices with OpenTelemetry: propagate trace context, instrument clients, and avoid common pitfalls.Developer toolsExporting LLM traces to Jaeger with OpenTelemetryLearn how exporting LLM traces to Jaeger with OpenTelemetry works: stand up Jaeger, instrument Python LLM calls, and verify spans end to end.Developer toolsInstrumenting LLM calls with OpenTelemetry spansLearn how to implement instrumenting LLM calls with OpenTelemetry spans step by step, capturing latency, token usage, and errors across your LLM stack.Developer toolsMeasuring token counts as OpenTelemetry span attributesLearn how to emit token counts as OpenTelemetry span attributes for LLM apps, with runnable Python code for instrumentation and verification.Developer toolsOpenTelemetry context propagation for async LLM callsLearn how to implement OpenTelemetry context propagation async patterns for LLM apps, keeping traces intact across asyncio tasks and threads.Developer toolsOpenTelemetry semantic conventions for GenAI, explainedA precise explainer of OpenTelemetry GenAI semantic conventions: what they define, how spans model LLM calls, and why standardized tracing matters.

All opentelemetry tracing for llm apps posts →