Exporting LLM traces to Jaeger with OpenTelemetry

Exporting LLM traces to Jaeger with OpenTelemetry gives you a single pane of glass for latency, token usage, and error paths across model calls. This walkthrough instruments a Python service that calls an OpenAI-compatible chat endpoint, exports spans over OTLP, and renders them in Jaeger. You’ll get a reproducible pattern you can drop into any LLM app, regardless of which backend model serves the response.

Step 1: Start Jaeger with OTLP ingestion

Jaeger’s all-in-one image bundles the collector, query, and UI. Enable the OTLP receiver explicitly; older images default to off.

docker run --rm -d \
  --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/all-in-one:1.57

Port 4317 is the OTLP gRPC endpoint. Port 16686 is the web UI. Verify the container is healthy:

curl -s localhost:16686/api/services

Expect {"data":[],"total":0,"limit":0,"offset":0,"errors":null} before any traces land. If you see a connection refused, the container isn’t up or the port mapping is wrong.

Step 2: Install the OpenTelemetry Python SDK

Use the gRPC OTLP exporter. Pin versions to avoid silent breakage from upstream releases.

pip install opentelemetry-api==1.24.0 \
            opentelemetry-sdk==1.24.0 \
            opentelemetry-exporter-otlp-proto-grpc==1.24.0 \
            openai==1.30.0

The grpcio dependency comes transitively. On M1/M2 Macs, you may need grpcio precompiled wheels; if install fails, run pip install grpcio==1.62.0 first.

Step 3: Configure the tracer provider

Set a global TracerProvider with a BatchSpanProcessor writing to the local OTLP endpoint. Batching decouples trace emission from network calls so your LLM latency isn’t polluted by export blocking.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
otlp = OTLPSpanExporter(endpoint="localhost:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(otlp))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("llm.app")

The string "llm.app" becomes the service.name in Jaeger unless you override it via a Resource. For local dev, insecure=True skips TLS. In production, point endpoint at your collector’s domain and remove that flag.

You can also drive the endpoint from env vars, which is cleaner for deploys:

import os
otlp = OTLPSpanExporter(
    endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "localhost:4317"),
    insecure=True,
)

Step 4: Wrap the LLM call in a span

Create a span around the completion call. Record model name, token counts, and latency as attributes. If you use an OpenAI-compatible client, the same code works regardless of the backend model.

import time
from openai import OpenAI

client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")

def chat(prompt: str, model: str = "gpt-4o-mini"):
    with tracer.start_as_current_span("llm.chat") as span:
        span.set_attribute("llm.model", model)
        span.set_attribute("llm.prompt_chars", len(prompt))
        start = time.perf_counter()
        try:
            resp = client.chat.completions.create(
                model=model,
                messages=[{"role": "user", "content": prompt}],
            )
        except Exception as e:
            span.record_exception(e)
            span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
            raise
        elapsed = time.perf_counter() - start
        span.set_attribute("llm.latency_ms", round(elapsed * 1000, 2))
        span.set_attribute("llm.completion_tokens", resp.usage.completion_tokens)
        span.set_attribute("llm.prompt_tokens", resp.usage.prompt_tokens)
        span.set_attribute("llm.total_tokens", resp.usage.total_tokens)
        return resp.choices[0].message.content

If you’re routing through a gateway such as n4n.ai—which exposes one OpenAI-compatible endpoint across 240+ models with automatic fallback when a provider is degraded—the base_url change is the only diff; the span attributes stay identical. That’s the payoff of instrumenting at the client boundary instead of per-provider.

Context propagation

For async chains, pass the span context explicitly. OpenTelemetry’s contextvars integration travels across asyncio tasks if you use tracer.start_as_current_span inside the coroutine. Don’t rely on thread-local state in event loops.

Step 5: Generate traffic and flush

Call the function a few times, then shut down the provider to flush buffered spans.

if __name__ == "__main__":
    print(chat("Explain OTel span processors in one sentence."))
    print(chat("What is the OTLP default gRPC port?"))
    provider.shutdown()

The BatchSpanProcessor exports on a timer (default 5s), but provider.shutdown() forces a flush before process exit. Without it, the last traces may be lost.

For async code, use asyncio.run and call provider.shutdown() in the finally block:

import asyncio

async def main():
    loop = asyncio.get_event_loop()
    await loop.run_in_executor(None, chat, "async test")

asyncio.run(main())
provider.shutdown()

Step 6: Verify traces in Jaeger

Open http://localhost:16686. In the Service dropdown, select llm.app. Click Find Traces.

After exporting LLM traces to Jaeger with OpenTelemetry, you should see two traces, each with a llm.chat span. Expand a span to inspect attributes:

llm.model
llm.latency_ms
llm.total_tokens

If nothing appears within ten seconds, check the exporter logs. Common failures: wrong port (4317 for gRPC, 4318 for HTTP), or the Jaeger container missing COLLECTOR_OTLP_ENABLED=true. Also confirm the process didn’t exit before the batch timer fired—always call shutdown().

Query by attribute

Jaeger supports key-value search. Enter llm.model=gpt-4o-mini to filter. This is why we set explicit attributes rather than stuffing data into span names; names are for flow, attributes are for query.

Step 7: Add automatic HTTP instrumentation (optional)

The LLM client makes HTTPS calls under the hood. Capture them with the OTel requests instrumentation to see DNS, TLS, and request size as child spans.

pip install opentelemetry-instrumentation-requests==0.45b0

from opentelemetry.instrumentation.requests import RequestsInstrumentor
RequestsInstrumentor().instrument()

Now each llm.chat span has a child HTTP POST span. This reveals whether latency is in your code or the network. If you use httpx instead, install opentelemetry-instrumentation-httpx.

Step 8: Production considerations

Sampler: default is always-on. For high traffic, use TraceIdRatioBased(0.1) to keep 10% of traces.
Resource attributes: set service.name and deployment.environment so Jaeger groups correctly.
Batching: tune ScheduleDelayMillis on BatchSpanProcessor if you see export lag.
Token truncation: never put full prompts in span attributes in prod; use a log exporter or a dedicated eval store.
Status codes: set span.set_status on error so Jaeger marks the trace failed.

Example resource config:

from opentelemetry.sdk.resources import Resource
resource = Resource.create({
    "service.name": "llm-proxy",
    "deployment.environment": "staging",
})
provider = TracerProvider(resource=resource)

Verify success checklist

Jaeger UI lists llm.app service after a run.
Each LLM call shows a span with token attributes.
HTTP child span appears when requests instrumentation is on.
provider.shutdown() returns without error and spans are visible within 5s.
Error paths show red spans when you force a bad API key.

Exporting LLM traces to Jaeger with OpenTelemetry is mostly wiring the SDK correctly; the value comes from consistent attribute naming. Once this pipeline is stable, you can swap models or providers without losing observability, and your on-call engineers get real data instead of guesswork.