What is LLM observability? It is the discipline of collecting, correlating, and analyzing telemetry from every interaction with a language model—prompts, completions, tool calls, retries, and latency—so you can debug and optimize systems in production. Unlike traditional application performance monitoring, it must capture the unstructured inputs and outputs that determine model behavior, not just status codes and stack traces.
How LLM observability works
At its core, LLM observability stitches together three signals: traces, metrics, and logs. The trace is the spine. Each user request that triggers one or more model calls becomes a distributed trace with spans for the orchestration code, the gateway hop, and the provider inference.
Tracing the request path
A span should record the model name, the endpoint, the number of input and output tokens, and the latency breakdown. If you use an OpenAI-compatible client, you can wrap the call:
from openai import OpenAI
from opentelemetry import trace
tracer = trace.get_tracer("llm.app")
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="sk-...")
def chat(prompt: str):
with tracer.start_as_current_span("llm.call") as span:
span.set_attribute("llm.model", "gpt-4o-mini")
span.set_attribute("llm.prompt_chars", len(prompt))
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
)
span.set_attribute("llm.completion_tokens", resp.usage.completion_tokens)
span.set_attribute("llm.prompt_tokens", resp.usage.prompt_tokens)
return resp.choices[0].message.contentThe snippet above points at an OpenAI-compatible endpoint. n4n.ai exposes one such endpoint across 240+ models and forwards provider cache-control hints, so the span attributes stay accurate even when the underlying provider changes.
Capturing payloads and metadata
Traces without the actual prompt and completion are useless for debugging. You need to store the text (or a hashed variant for privacy) as span events or linked log records. Keep the temperature, top_p, and any seed values as attributes—these are the knobs that explain why two identical prompts produced different outputs.
Sampling matters here. At high traffic you cannot retain every full payload. Use head sampling for health metrics and tail sampling to keep traces where the model returned an error or exceeded a token budget. Never drop the token counts; they are cheap and high-value.
Metrics and logs
Aggregate token counts per model, per route, and per user. Watch p50/p95 latency and error rates. Logs should include the raw exception from the provider, including rate-limit headers. When a provider is degraded, an inference gateway that performs automatic fallback will shift traffic; your metrics must reflect which provider actually served the token.
Per-token metering is the only way to attribute cost precisely. If your gateway already emits usage events, pipe them into the same time series as your spans to close the loop.
Why it matters
Debugging nondeterministic failures
A retrieval-augmented generation (RAG) pipeline returns a confident but wrong answer. Without observability, you guess whether the retriever failed, the prompt misformatted the context, or the model ignored instructions. With a trace, you see the retrieved chunks, the exact prompt sent, and the completion. You fix the prompt, not the vector DB.
Cost and latency control
Token spend is the dominant cost. Per-token metering lets you attribute cost to a feature or a customer. If a single agent loop burns 40k tokens because a tool returned a huge JSON, the trace shows the loop span and the offending tool call. You cap it or paginate.
Regulatory and eval needs
Many teams must show what was sent to a model for a given user action. Observability stores the audit trail. It also feeds evaluation harnesses: you replay traces to measure regression after a model swap. This is what is LLM observability buys you beyond incident response—a reusable dataset of real interactions.
A concrete example
Consider a support agent that calls a search tool, then a summarizer, then a classifier. A user reports the agent gave a refund policy that doesn’t exist.
Instrument the agent
def run_agent(query):
with tracer.start_as_current_span("agent") as root:
root.set_attribute("user.id", query.user_id)
docs = search_tool(query.text) # span: tool.search
summary = summarize(docs) # span: llm.summarize
label = classify(summary) # span: llm.classify
return label, summaryAnalyze the trace
The trace reveals tool.search returned docs from 2022. The llm.summarize span shows the prompt included those stale docs. The classifier then echoed the stale refund rule. The fix is a date filter in the search tool, not a model change.
A simplified span in JSON might look like:
{
"name": "llm.summarize",
"attributes": {
"llm.model": "claude-3-5-haiku",
"llm.prompt_tokens": 1820,
"llm.completion_tokens": 240,
"tool.search.hits": 3,
"tool.search.newest_year": 2022
},
"events": [
{"name": "prompt", "attributes": {"text": "Summarize: ...[stale docs]..."}}
]
}This is what is LLM observability in practice: you pinpoint the fault without redeploying or adding print statements.
Common misconceptions
“Logging the prompt is enough”
Raw logs are unstructured and uncorrelated. You cannot answer “what was the p95 latency for the summarize step across the last 24 hours?” from a grep of stdout. Structured spans give you aggregation.
“It’s just API monitoring”
API monitoring tells you the /v1/chat/completions endpoint returned 429. It does not tell you which user, which chain step, or what prompt caused the retry storm. LLM observability maps the call into your application’s control flow.
“Only needed for big teams”
A solo developer shipping a CLI tool still needs to know why a call cost $0.50 instead of $0.01. Traces pay off at day one.
“Observability slows down inference”
A well-designed SDK wrapper adds microseconds of attribute setting. The network round-trip to the model dominates. If you batch span export asynchronously, the user sees no difference.
Minimum viable setup
You do not need a commercial platform to start. Use OpenTelemetry with the Python or TypeScript SDK, export to a local Jaeger instance, and wrap your LLM client. Capture at least: model, prompt/completion tokens, latency, and the prompt text.
# minimal async exporter setup
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")))
trace.set_tracer_provider(provider)Once this is in place, every model call becomes debuggable. As your system grows, you can swap the exporter for a managed backend or route through a gateway that already meters tokens.
Sampling and privacy
Treat prompt and completion text as sensitive data. Hash or truncate before sending to a third-party collector unless you have a data processing agreement. Use deterministic sampling on high-volume low-value paths (health checks, trivial classifications) and full retention on error and high-token spans.
Correlation IDs are mandatory. Propagate a trace_id from the HTTP request through to the model call so support tickets can be tied to a specific inference.
Relationship to evaluation
Observability and evaluation are cousins. The trace store is the cheapest source of real-world prompts and completions for building eval sets. Export a sample of production traces weekly, strip PII, and use them as regression tests when you switch models or tweak prompts.
Where the gateway fits
If you already route inference through an OpenAI-compatible gateway, you can inherit telemetry without instrumenting every call. For example, n4n.ai provides per-token usage metering and honors client routing directives, so your spans can be enriched with the actual provider used after fallback. That removes a class of missing-data bugs when providers degrade.
But the gateway does not replace application-level tracing. You still need to record the business context—the user, the session, the tool outputs—that only your code knows.
Closing checklist
- Emit a span per model call with token counts and model ID.
- Store prompt and completion as events, not just attributes, to avoid size limits.
- Track fallback provider changes if you use automatic routing.
- Alert on token burn rate per route, not just error rate.
- Replay traces in evals when you change models.
What is LLM observability if not the difference between shipping blind and shipping with a flight recorder? Treat it as a first-class concern, not an afterthought. Build the flight recorder before you need it.