Multi-step LLM chains fail in ways monolithic apps never do. Distributed tracing for LLM chains is the only way to see which prompt, tool call, or model hop ate 30 seconds or silently degraded output.
The failure modes of multi-step LLM chains
A chain that calls an LLM, parses JSON, calls a search tool, then calls the LLM again introduces at least four discrete failure points. Any step can hang, return malformed data, or consume an order of magnitude more tokens than expected.
In a single LLM call, you can reason about latency from the provider’s dashboard. In a chain, the user-facing delay is the sum of orchestration overhead, network round-trips, and cumulative generation time. When the final answer is wrong, the bug might live in the second prompt’s system message, not the model itself.
Compounding cost is another mode. A retrieval-augmented generation (RAG) flow that embeds a query, fetches 20 documents, and stuffs them into a context can blow past token budgets silently. Without per-step attribution you will see a large bill but not the culprit.
Agents that loop until a condition is met are worse. A poorly designed stopping criterion can trigger 15 model calls before timing out. The user experiences a 90-second hang; your logs show 15 separate “completed” lines with no indication of the loop.
Why logs are insufficient
Logs are local and unstructured by default. A typical Python service logs “calling LLM” and later “got response”. Correlating those lines across an async worker that runs the tool call requires a shared ID you manually thread through every function.
Even with a request ID, logs don’t model causal hierarchy. You cannot ask “what was the critical path of this chain?” from a grep. You also cannot see the parent-child relationship between the orchestrator span and the nested embedding call without explicit instrumentation.
Distributed tracing for LLM chains solves this by making the causal graph first-class. Each operation is a span with a start, end, and parent. The trace is the tree of everything that happened to serve one user request.
Distributed tracing for LLM chains: the mental model
A trace is a directed acyclic graph of spans. For an LLM chain, useful spans include:
chain.execute— the top-level orchestration.llm.generate— a single model call, with attributes for model name, token counts, and finish reason.tool.invoke— an external API or function call.retrieval.query— vector search or SQL lookup.
The key is context propagation. When the orchestrator spawns a background task for a tool, it must pass the traceparent header or equivalent context so the child span links correctly.
Spans, context, and attributes
OpenTelemetry defines semantic conventions for AI workloads (currently evolving under gen_ai.*). At minimum, set:
gen_ai.system(e.g., “openai”)gen_ai.request.modelgen_ai.usage.prompt_tokensgen_ai.usage.completion_tokens
Avoid putting full prompt text in span attributes by default; it can leak PII. Use a redacted summary or rely on a separate log sink with strict access control.
Minimal Python instrumentation
Below is a working pattern using the OpenTelemetry SDK and the standard OpenAI client. It creates a span per LLM call and records token usage from the response object.
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
import openai
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("rag_chain")
def generate(messages, model="gpt-4o-mini"):
with tracer.start_as_current_span("llm.generate") as span:
span.set_attribute("gen_ai.system", "openai")
span.set_attribute("gen_ai.request.model", model)
resp = openai.chat.completions.create(model=model, messages=messages)
span.set_attribute("gen_ai.usage.prompt_tokens", resp.usage.prompt_tokens)
span.set_attribute("gen_ai.usage.completion_tokens", resp.usage.completion_tokens)
span.set_attribute("gen_ai.response.finish_reasons", [resp.choices[0].finish_reason])
return respThis is enough to see latency and token counts in a local collector. For production, swap ConsoleSpanExporter for OTLP to a Collector or vendor.
Example trace shape
A serialized trace makes the value obvious:
{
"traceId": "a1b2c3",
"spans": [
{ "name": "chain.execute", "durationMs": 4200 },
{ "name": "retrieval.query", "durationMs": 300, "attributes": { "docs_returned": 20 } },
{ "name": "llm.generate", "durationMs": 3900, "attributes": { "gen_ai.request.model": "gpt-4o-mini", "gen_ai.usage.prompt_tokens": 3100 } }
]
}The bottleneck is immediately visible: the LLM call dominates, and the retrieval cost is negligible.
Cross-provider routing and fallback
Many teams abstract model access behind an OpenAI-compatible gateway to avoid vendor lock-in. When a gateway like n4n.ai automatically fails over between providers on rate limits, the trace must record which backend actually served the request. Otherwise you will blame “the model” for a latency spike that was a secondary provider’s cold start.
Propagate the W3C traceparent header on every outbound call. Capture the resolved provider from a response header or body field and attach it to the span. If the gateway forwards provider cache-control hints, record cache hits to explain zero token charges.
const span = tracer.startSpan("llm.generate");
const traceParent = span.spanContext().traceParent;
const res = await fetch("https://api.n4n.ai/v1/chat/completions", {
method: "POST",
headers: {
"content-type": "application/json",
"traceparent": traceParent,
},
body: JSON.stringify({ model: "auto", messages }),
});
const data = await res.json();
span.setAttribute("gen_ai.response.provider", res.headers.get("x-provider") ?? "unknown");
span.setAttribute("gen_ai.cache.hit", res.headers.get("x-cache") === "HIT");
span.end();The gateway’s per-token usage metering then aligns with your span’s token attributes, giving you a single source of truth for cost.
Tradeoffs you must accept
Tracing is not free. Each span has creation overhead and the exporter adds network egress. At high request volumes, you will implement sampling.
Head-based sampling (decide at trace start) is simple but may drop the exact slow chain you needed. Tail-based sampling (decide after the trace completes, in the Collector) preserves errors and high-latency paths but requires buffering and more infrastructure.
Another tradeoff is instrumentation coverage. If one internal tool call is not wrapped, the trace shows a gap that looks like idle time. You must discipline your team to wrap every I/O boundary.
Privacy is the third cost. Prompts and completions are data. You should default to not exporting full content, and use deterministic redaction if you need visibility. In regulated environments, export only hashes of prompt prefixes.
Finally, trace storage is not cheap. A month of detailed chain traces for a busy agent product can reach terabytes. Set retention policies and aggregate metrics (p95 latency per span name) before raw trace cost surprises you.
What to instrument first
If you are retrofitting, start with the top-level chain span and the primary LLM call. That alone answers “which model and how many tokens” for the critical path.
Next, wrap tool invocations with input/output size and latency. Then add retrieval spans with the number of documents returned. Within a day you will have enough to diagnose the majority of incidents.
Adopt the OpenTelemetry Collector with OTLP ingestion. It decouples your app from the backend, letting you switch from Jaeger to a commercial trace store without code changes. Configure batch processing to amortize export cost.
Takeaway
Distributed tracing for LLM chains is not optional once you move past a single prompt-response call. It converts mysterious latency and cost spikes into a navigable causal graph. Instrument with OpenTelemetry, propagate context across every provider hop, and sample to balance cost against debuggability.
Ship tracing from the first commit of your chain code. Retrofitting after a production incident will cost more than the span overhead you avoided.