n4nAI
Open console →
Developer toolsTutorialOpenTelemetry Tracing for LLM Apps

Adding OpenTelemetry to a Python LLM app in 20 minutes

Learn to add OpenTelemetry to a Python LLM app in 20 minutes with this hands-on tutorial covering spans, attributes, and Jaeger output.

n4n Team3 min read561 words

Audio narration

Coming soon — every post will get a voice note here.

If you’re building an OpenTelemetry Python LLM app, you need visibility into where latency and token cost accumulate across prompt assembly, model inference, and output parsing. This tutorial takes a bare Python script that calls an LLM and adds proper distributed tracing with OpenTelemetry in about 20 minutes, using the standard OTel SDK and a local Jaeger backend.

Prerequisites

  • Python 3.10 or newer
  • Docker (for Jaeger all-in-one)
  • An OpenAI-compatible endpoint. You can use OpenAI directly, or route through a gateway like n4n.ai that exposes one OpenAI-compatible endpoint for 240+ models with automatic fallback when a provider is degraded.
  • pip and a virtual environment

Install the required packages:

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp openai

Step 1: Run Jaeger locally

Jaeger’s all-in-one image ships with an OTLP gRPC receiver on port 4317 and a UI on 16686.

docker run --rm -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:latest

Open http://localhost:16686. The UI loads with an empty trace list. Keep this terminal running.

Step 2: Configure the TracerProvider

Create telemetry.py. This module initializes the SDK once and exposes a tracer.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "service.name": "llm-app",
    "service.version": "0.1.0",
})

provider = TracerProvider(resource=resource)
otlp = OTLPSpanExporter(endpoint="localhost:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(otlp))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("llm-app.tracer")

The BatchSpanProcessor buffers spans and exports them on a background thread. Never use SimpleSpanProcessor in production; it blocks on every span.

Step 3: Make a traced LLM call

We wrap the OpenAI client. The openai library accepts a base_url, so pointing it at any OpenAI-compatible gateway works without code changes.

import openai
from opentelemetry import trace
from telemetry import tracer

# Example using an OpenAI-compatible gateway
client = openai.OpenAI(base_url="https://api.n4n.ai/v1", api_key="dummy-key")

def generate(prompt: str, model: str = "gpt-4o-mini") -> str:
    with tracer.start_as_current_span("llm.generate") as span:
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.prompt", prompt)
        span.set_attribute("gen_ai.system", "openai-compatible")
        try:
            resp = client.chat.completions.create(
                model=model,
                messages=[{"role": "user", "content": prompt}],
            )
            completion = resp.choices[0].message.content
            span.set_attribute("gen_ai.completion", completion)
            span.set_attribute("gen_ai.usage.prompt_tokens", resp.usage.prompt_tokens)
            span.set_attribute("gen_ai.usage.completion_tokens", resp.usage.completion_tokens)
            return completion
        except Exception as e:
            span.record_exception(e)
            span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
            raise

Because n4n.ai honors client routing directives and forwards provider cache-control hints, the same span attributes remain accurate when the gateway fails over to a secondary provider mid-request.

Step 4: Run the script and verify in Jaeger

Create main.py:

from telemetry import tracer
from llm import generate

with tracer.start_as_current_span("app.run") as root:
    root.set_attribute("app.input", "Explain OTel in one line")
    out = generate("Explain OpenTelemetry in one sentence.")
    print(out)

Run it:

python main.py

The console prints the model response. Within a few seconds, the BatchSpanProcessor flushes to Jaeger. In the UI, select service llm-app and click Find Traces. You’ll see one trace containing app.run as the root and llm.generate nested inside. Expand the child span to inspect attributes:

  • gen_ai.request.model: gpt-4o-mini
  • gen_ai.usage.prompt_tokens: integer
  • gen_ai.usage.completion_tokens: integer

That’s the minimum viable OpenTelemetry Python LLM app.

Step 5: Model a real chain with multiple spans

Single calls are rare. Add a retrieval step that builds context before generation.

import time
from telemetry import tracer

def retrieve(query: str) -> list[str]:
    with tracer.start_as_current_span("retrieve") as span:
        span.set_attribute("retrieval.query", query)
        start = time.perf_counter()
        docs = ["OTel is a CNCF standard for telemetry.", "Spans represent units of work."]
        time.sleep(0.05)  # simulate vector search
        span.set_attribute("retrieval.latency_ms", (time.perf_counter() - start) * 1000)
        span.set_attribute("retrieval.count", len(docs))
        return docs

Update main.py to compose them:

from telemetry import tracer
from llm import generate
from retrieve import retrieve

with tracer.start_as_current_span("app.run") as root:
    docs = retrieve("What is OpenTelemetry?")
    prompt = f"Context: {docs}\nAnswer the question."
    out = generate(prompt)
    print(out)

The resulting trace shows three spans in sequence: app.run → retrieve → llm.generate. Nested spans make it obvious which stage dominates latency.

Step 6: Instrument prompt construction

Prompt building often involves templating and string concatenation that can hide cost. Wrap it:

from telemetry import tracer

def build_prompt(docs: list[str], question: str) -> str:
    with tracer.start_as_current_span("prompt.build") as span:
        span.set_attribute("prompt.doc_count", len(docs))
        template = "Context: {ctx}\nQ: {q}\nA:"
        prompt = template.format(ctx=" ".join(docs), q=question)
        span.set_attribute("prompt.length", len(prompt))
        return prompt

Call it between retrieve and generate. Now you have four spans and can attribute token bloat to specific template changes.

Step 7: Context propagation across threads

If you push LLM calls into a concurrent.futures.ThreadPoolExecutor, the current span context must be captured manually. OTel uses contextvars under the hood.

from opentelemetry.context import get_current, attach, detach

def submit_to_pool(pool, prompt):
    ctx = get_current()
    def worker():
        token = attach(ctx)
        try:
            return generate(prompt)
        finally:
            detach(token)
    return pool.submit(worker)

Without attach/detach, the worker span becomes a new trace root, breaking the chain.

Step 8: Production adjustments

  • Sampling: default is AlwaysOnSampler. Set a ParentBasedTraceIdRatioSampler at 10% for high-volume apps.
  • Export: point OTLPSpanExporter at an OpenTelemetry Collector, not directly at Jaeger, so you can add batching and retries.
  • Semantic conventions: the gen_ai.* keys are still evolving; pin a version in your codebase and document it.
  • Cost mapping: if your gateway reports per-token usage, the gen_ai.usage.* attributes let you compute spend per trace in a backend dashboard.

Expected exported span (abbreviated)

{
  "name": "llm.generate",
  "attributes": {
    "gen_ai.request.model": "gpt-4o-mini",
    "gen_ai.system": "openai-compatible",
    "gen_ai.prompt": "Context: ...",
    "gen_ai.completion": "OpenTelemetry is an open standard...",
    "gen_ai.usage.prompt_tokens": 24,
    "gen_ai.usage.completion_tokens": 11
  },
  "parent_span_id": "9a3f...",
  "trace_id": "1b4e..."
}

You now have a working OpenTelemetry Python LLM app with end-to-end traces, token usage attributes, and a local Jaeger instance to explore them. From here, swap the dummy key for a real one, point the exporter at your collector, and add spans to every stage of your pipeline.

Tagsopentelemetrypythontutorialllm-apps

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Developer toolsExporting LLM traces to Jaeger with OpenTelemetryLearn how exporting LLM traces to Jaeger with OpenTelemetry works: stand up Jaeger, instrument Python LLM calls, and verify spans end to end.Developer toolsInstrumenting LLM calls with OpenTelemetry spansLearn how to implement instrumenting LLM calls with OpenTelemetry spans step by step, capturing latency, token usage, and errors across your LLM stack.Developer toolsOpenTelemetry collector configs for high-volume LLM trafficPractical OpenTelemetry collector configurations for scaling LLM tracing and metrics, with pipeline patterns, batching, and sampling for high-volume inference.Developer toolsSending OpenTelemetry LLM traces to Grafana TempoStep-by-step tutorial to send OpenTelemetry LLM traces to Grafana Tempo: run Tempo with Docker, instrument a Python LLM client, query spans.Developer toolsTracing a RAG pipeline end to end with OpenTelemetryLearn how to implement tracing RAG with OpenTelemetry in Python: instrument retrieval, augmentation, and generation steps with runnable code and spans.Developer toolsCross-service LLM span correlation with OpenTelemetryA practical guide to correlating LLM spans across microservices with OpenTelemetry: propagate trace context, instrument clients, and avoid common pitfalls.

All opentelemetry tracing for llm apps posts →