n4nAI
Open console →
Developer toolsHow-toOpenTelemetry Tracing for LLM Apps

Instrumenting LLM calls with OpenTelemetry spans

Learn how to implement instrumenting LLM calls with OpenTelemetry spans step by step, capturing latency, token usage, and errors across your LLM stack.

n4n Team3 min read708 words

Audio narration

Coming soon — every post will get a voice note here.

Instrumenting LLM calls with OpenTelemetry gives you latency breakdowns, token counts, and error traces across your inference stack. This guide walks through a concrete setup using the OpenTelemetry Python SDK and a typical OpenAI-compatible client, so you can see exactly where time goes when you call a model.

Step 1: Install and initialize the OpenTelemetry SDK

When instrumenting LLM calls with OpenTelemetry, start by installing the API, SDK, and an OTLP exporter. For local development the HTTP OTLP exporter pointed at a local collector is the most realistic path; the console exporter works for a quick shape check.

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

Initialize a TracerProvider with a Resource that identifies your service and environment. Export to a local collector endpoint.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "service.name": "llm-proxy-client",
    "service.version": "1.0.0",
    "deployment.environment": "dev",
})
provider = TracerProvider(resource=resource)
exporter = OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("llm.client")

The BatchSpanProcessor buffers spans and flushes on a timer. For a CLI script that exits fast, call provider.force_flush() before exit, or you will lose the last batch.

Step 2: Wrap your LLM call in a span

The core of instrumenting LLM calls with OpenTelemetry is creating a span around the network request and recording input metadata. Use start_as_current_span so any internal retries or child HTTP calls nest under it.

from openai import OpenAI

client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")

def chat_with_span(messages, model="gpt-4o-mini"):
    with tracer.start_as_current_span("llm.chat") as span:
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.request.temperature", 0.7)
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.prompt.length", sum(len(m["content"]) for m in messages))

        response = client.chat.completions.create(
            model=model,
            messages=messages,
            temperature=0.7,
        )
        return response

The gen_ai.* attribute keys follow the OpenTelemetry semantic conventions for generative AI. They let observability backends group by model or operation without custom parsing. Avoid logging full prompt text as an attribute; it balloons span size and risks leaking PII.

TypeScript equivalent

In Node, use @opentelemetry/sdk-node:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { trace } from '@opentelemetry/api';

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({ url: 'http://localhost:4318/v1/traces' }),
  serviceName: 'llm-proxy-client',
});
sdk.start();

const tracer = trace.getTracer('llm.client');

async function chat(messages: any[], model = 'gpt-4o-mini') {
  return tracer.startActiveSpan('llm.chat', async (span) => {
    span.setAttribute('gen_ai.request.model', model);
    const res = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      headers: { 'content-type': 'application/json', authorization: `Bearer ${process.env.OPENAI_KEY}` },
      body: JSON.stringify({ model, messages, temperature: 0.7 }),
    });
    const json = await res.json();
    span.end();
    return json;
  });
}

Step 3: Record token usage and latency

A span without outcome data is just a timer. Extract usage from the response and set attributes before the span closes. Capture exceptions so failures are visible in the trace.

def chat_with_span(messages, model="gpt-4o-mini"):
    with tracer.start_as_current_span("llm.chat") as span:
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.operation.name", "chat")
        try:
            response = client.chat.completions.create(
                model=model,
                messages=messages,
                temperature=0.7,
            )
            usage = response.usage
            span.set_attribute("gen_ai.usage.prompt_tokens", usage.prompt_tokens)
            span.set_attribute("gen_ai.usage.completion_tokens", usage.completion_tokens)
            span.set_attribute("gen_ai.response.id", response.id)
            span.set_status(trace.Status(trace.StatusCode.OK))
            return response
        except Exception as e:
            span.record_exception(e)
            span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
            raise

Latency is automatically captured by the span’s start and end timestamps. If you need to break down time-to-first-token for streaming, start a child span when the first chunk arrives.

Step 4: Handle streaming and async contexts

For streaming responses, the request returns quickly but generation runs over seconds. Instrumenting LLM calls with OpenTelemetry in streaming mode requires explicit child spans because the parent finishes before the user sees output.

import time

def stream_chat(messages, model="gpt-4o-mini"):
    with tracer.start_as_current_span("llm.chat.stream") as span:
        span.set_attribute("gen_ai.request.model", model)
        stream = client.chat.completions.create(model=model, messages=messages, stream=True)
        with tracer.start_as_current_span("llm.stream.consume") as child:
            start = time.time()
            first_token = None
            for chunk in stream:
                if first_token is None and chunk.choices[0].delta.content:
                    first_token = time.time()
                    child.set_attribute("gen_ai.stream.ttft_ms", (first_token - start) * 1000)
                yield chunk
            child.set_attribute("gen_ai.stream.total_tokens", chunk.usage.completion_tokens if hasattr(chunk, "usage") else -1)

In async Python, use tracer.start_as_current_span inside an async with block. The context propagation works the same; just ensure your HTTP client (e.g., httpx with openai.AsyncClient) is instrumented or at least shares the same event loop.

Step 5: Export to a collector and visualize

Run an OTLP-compatible collector. Jaeger all-in-one is the fastest local option:

docker run --rm -p 4318:4318 -p 16686:16686 jaegertracing/all-in-one:latest

Point your exporter at http://localhost:4318/v1/traces (already configured in Step 1). Generate a few calls, then open http://localhost:16686. You should see the llm.chat span with attributes like gen_ai.usage.prompt_tokens.

If you route through a gateway such as n4n.ai, its OpenAI-compatible endpoint forwards provider cache-control hints; record gen_ai.response.cache_hit from the response headers to distinguish cached completions in your traces. That single attribute tells you whether a fallback provider served the request.

Step 6: Add routing and fallback context

Production systems rarely call a single provider. When you route through a proxy that performs automatic fallback, add the resolved provider and attempt count as span attributes. This makes incidents debuggable: you can see that a timeout on provider A triggered a fallback to provider B.

span.set_attribute("routing.final_provider", response.headers.get("x-provider", "unknown"))
span.set_attribute("routing.attempts", int(response.headers.get("x-routing-attempts", 1)))

Gateways like n4n.ai emit per-token usage metering via the same endpoint; reconcile that with your gen_ai.usage.* span attributes to catch drift between traced and billed tokens. Honoring client routing directives means you can also record the requested provider versus the served provider to measure fallback frequency.

Verify success

Run your instrumented script against the collector. Check for these outcomes:

  1. The process exits without ValueError: No active span or export errors.
  2. In Jaeger, the llm-proxy-client service appears with llm.chat spans.
  3. Expanding a span shows gen_ai.request.model, gen_ai.usage.prompt_tokens, and gen_ai.usage.completion_tokens.
  4. Force a failure (bad API key) and confirm the span status is ERROR with an exception event.

If you used the console exporter, the JSON on stdout should contain "name": "llm.chat" and the attribute list. That is the minimal proof that instrumenting LLM calls with OpenTelemetry is wired correctly.

Caveats and next steps

Do not put raw prompt content in span attributes by default—it can leak PII into your trace backend. Use event bodies or a redaction processor if you need full payloads. For high-volume services, batch spans and sample at 10–20% using ParentBased samplers.

Once spans are flowing, build a dashboard that plots p95 token latency per model. That view will immediately show you which model versions are degrading and whether fallback routing is firing more than expected.

Tagsopentelemetrytracingllm-appsspans

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Developer toolsExporting LLM traces to Jaeger with OpenTelemetryLearn how exporting LLM traces to Jaeger with OpenTelemetry works: stand up Jaeger, instrument Python LLM calls, and verify spans end to end.Developer toolsAdding OpenTelemetry to a Python LLM app in 20 minutesLearn to add OpenTelemetry to a Python LLM app in 20 minutes with this hands-on tutorial covering spans, attributes, and Jaeger output.Developer toolsCross-service LLM span correlation with OpenTelemetryA practical guide to correlating LLM spans across microservices with OpenTelemetry: propagate trace context, instrument clients, and avoid common pitfalls.Developer toolsDebugging slow LLM chains using OpenTelemetry waterfallsLearn how to trace and debug LLM chains with OpenTelemetry waterfalls. Step-by-step instrumentation, span analysis, and latency fixes for Python apps.Developer toolsMeasuring token counts as OpenTelemetry span attributesLearn how to emit token counts as OpenTelemetry span attributes for LLM apps, with runnable Python code for instrumentation and verification.Developer toolsOpenTelemetry collector configs for high-volume LLM trafficPractical OpenTelemetry collector configurations for scaling LLM tracing and metrics, with pipeline patterns, batching, and sampling for high-volume inference.

All opentelemetry tracing for llm apps posts →