n4nAI
Open console →
Developer toolsGuideLLM Observability Platforms

Building a vendor-neutral LLM observability stack

A practical guide to building a vendor-neutral LLM observability stack with OpenTelemetry, avoiding lock-in while tracking cost, latency, and quality.

n4n Team4 min read806 words

Audio narration

Coming soon — every post will get a voice note here.

Most teams bolt on a proprietary dashboard the moment they ship their first LLM feature, then spend months trapped in someone else’s data model. A vendor-neutral LLM observability stack lets you own your traces, metrics, and evaluations while keeping the freedom to switch providers or self-host. This guide lays out an ordered path to build one with open standards and minimal lock-in.

1. Define the signals before writing instrumentation

Decide what you must observe, not what a vendor’s UI shows you. The non-negotiables for any LLM call are: model identity, token counts, end-to-end latency, error type, and a stable hash of the prompt template. Everything else is situational.

Skip logging raw user inputs by default. If you need them for debugging, hash or truncate, and route through a separate access-controlled pipeline. Storage cost for full prompt/response bodies grows faster than your inference bill, and the privacy exposure is real.

Tradeoff: less detail means slower root-causing. Mitigate by sampling full bodies at 1–5% rather than capturing all traffic.

2. Adopt OpenTelemetry as the transport layer

Do not invent a custom JSON schema and pipe it to a S3 bucket. OpenTelemetry (OTel) gives you a vendor-neutral wire format (OTLP), mature SDKs, and a collector that can fan out to any backend. The gen-ai semantic conventions are still evolving, but adopting them now saves a migration later.

Minimal Python setup:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4318/v1/traces"))
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("llm.app")

Run the collector separately. Your app code stays ignorant of whether traces land in Tempo, Jaeger, or a commercial service.

3. Instrument the LLM call as a single span

Wrap each completion request in a span. Use the draft gen_ai.* attribute keys so any OTel-compatible tool can interpret them. The code below works against any OpenAI-compatible endpoint.

from openai import OpenAI

client = OpenAI(base_url="https://any-openai-compatible-endpoint/v1")

with tracer.start_as_current_span("chat_completion") as span:
    span.set_attribute("gen_ai.system", "openai")
    span.set_attribute("gen_ai.request.model", "gpt-4o-mini")
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Summarize the incident"}]
    )
    span.set_attribute("gen_ai.response.usage.prompt_tokens", resp.usage.prompt_tokens)
    span.set_attribute("gen_ai.response.usage.completion_tokens", resp.usage.completion_tokens)
    span.set_attribute("gen_ai.response.finish_reasons", [resp.choices[0].finish_reason])

Common pitfall: creating a span per token stream chunk. Emit one span for the request, and if you stream, record completion tokens when the stream closes. High-cardinality spans will starve your collector.

4. Capture cost and routing without proprietary SDKs

Token counts are enough to compute cost if you maintain a local price table. Do not call a vendor’s billing API from your hot path. Record routing metadata as attributes so you can see fallback behavior.

{
  "gen_ai.request.model": "claude-3-5-sonnet",
  "gen_ai.response.usage.prompt_tokens": 1200,
  "gen_ai.response.usage.completion_tokens": 300,
  "gen_ai.routing.fallback": "primary-degraded",
  "cache_control.hit": true
}

If you route through a gateway like n4n.ai, the OpenAI-compatible endpoint returns per-token usage and forwards provider cache-control hints, so the attributes above map directly to response fields without extra parsing. Honoring client routing directives means you can also record which provider actually served the request.

5. Deploy a self-hosted collector and backend

The OTel collector decouples your app from storage. A minimal config that writes to Tempo and logs locally:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
exporters:
  tempo:
    endpoint: tempo:4317
  debug: {}
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [tempo, debug]

Run this in the same VPC as your inference path. The app only needs the collector URL; swapping Tempo for another backend is a collector config change, not a code change.

6. Model evaluations as child spans

Quality signals belong in the same trace as the generation. Create a child span for each evaluation so you can correlate a bad score with the exact model, prompt, and latency.

with tracer.start_as_current_span("eval.toxicity", parent=span) as eval_span:
    score = toxicity_classifier(resp.choices[0].message.content)
    eval_span.set_attribute("eval.toxicity.score", score)
    eval_span.set_attribute("eval.toxicity.model", "distilbert-toxic")

Pitfall: running heavy evaluators synchronously inside the request path. Sample 10% of traffic for full eval, or push the eval span creation to a background worker that reads the completed trace. Blocking on a 200ms classifier per call will dominate your p99.

7. Query and alert on your own terms

With traces in Tempo and metrics in Prometheus (via the collector’s metrics pipeline), you can build Grafana dashboards that group by gen_ai.request.model or gen_ai.routing.fallback. A useful starting alert: p95 latency per model exceeds 2× the previous week’s baseline, or completion token count spikes without prompt change (possible loop or injection).

Avoid the trap of building a second proprietary UI because the raw traces feel too low-level. Invest in three or four dashboards; anything deeper is a notebook query.

8. Common pitfalls and tradeoffs

  • Full-body logging: defaults to “on” in many SaaS SDKs. Turn it off at the SDK level, not just in the UI.
  • Cardinality explosion: putting user IDs or request IDs as span attributes destroys indexing. Use them as span events or logs instead.
  • Eval coupling: if your eval library writes to its own database, you lose the trace correlation. Force it through OTel or accept the split.
  • Semantic drift: inventing your own model_name key instead of gen_ai.request.model means future tooling won’t understand you. Use the convention even if it feels verbose.

9. Keep the exit door unlocked

The point of a vendor-neutral LLM observability stack is that no single component owns your data shape. Your application emits OTLP. Your collector decides destination. Your price table lives in code or config. If a model provider goes down, you change a routing header; if a dashboard vendor raises prices, you point the collector elsewhere.

Build the stack once with these boundaries, and the next LLM you adopt is just another gen_ai.request.model value—not a rewrite.

Tagsllm-observabilityopentelemetryvendor-neutralarchitecture

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Developer toolsHow Helicone's proxy architecture captures LLM trafficEngineer's analysis of Helicone proxy architecture: how it intercepts LLM API traffic, tradeoffs vs SDK instrumentation, and when to use it.Developer tools8 LLM observability platforms compared for 2026A practitioner's breakdown of eight LLM observability platforms compared for 2026, covering deployment, tracing, cost, and eval tradeoffs for engineers.Developer toolsArize Phoenix for open-source LLM evaluation and tracingA practical guide to setting up Arize Phoenix open-source tracing for LLM evaluation and debugging, with code samples and production tradeoffs.Developer toolsArize Phoenix vs W&B Weave: tracing LLM apps head to headArize Phoenix vs W&B Weave compared across tracing capabilities, cost, latency, ergonomics, and ecosystem to help engineers pick an LLM observability stack.Developer toolsFive LangSmith alternatives worth evaluating in 2026A practitioner's comparison of five LangSmith alternatives for LLM observability in 2026, covering Langfuse, Helicone, Phoenix, AgentOps, and OpenLLMetry.Developer toolsHelicone vs Langfuse for LLM observabilityHelicone vs Langfuse compared across capabilities, pricing, latency, ergonomics, ecosystem, and limits to help engineers pick the right LLM observability stack.

All llm observability platforms posts →