How to choose an LLM observability platform for your team

Choosing an LLM observability platform is a decision that quietly shapes your team’s debugging speed and cost visibility for years. The wrong pick locks you into a vendor schema that doesn’t match how you actually call models, or hides the latency breakdown behind a pretty dashboard. This guide gives you an end-to-end process for choosing an LLM observability platform by running a real evaluation spike, not reading feature matrices.

Step 1: Map the spans you actually need

Before installing anything, write down the exact call graph of one production flow. For most teams that’s: a retriever call, a prompt assembly, a model completion, and optional tool loops. When choosing an LLM observability platform, span fidelity matters more than chart colors.

If you use agentic patterns, add spans for each tool invocation. A platform that only shows “LLM call” as a black box will waste your time later. You also need to decide what metadata to attach: user_id, session_id, and a release version are non-negotiable for slicing traces later.

Verify

List three flows on a whiteboard or doc. If a candidate tool can’t represent tool calls as nested spans in its demo, cut it.

Step 2: Stand up a candidate instance

Don’t argue about managed vs self-hosted yet. Spin up Langfuse locally via docker compose—it’s MIT licensed and gives you the full feature set. It needs a Postgres backend, which the compose file provides.

git clone https://github.com/langfuse/langfuse.git
cd langfuse
docker compose up -d

For LangSmith, create a project and export the keys:

export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=your_key
export LANGCHAIN_PROJECT=eval-spike

You now have two real environments to compare: one self-hosted, one managed.

Verify

Hit the Langfuse UI at localhost:3000 and confirm you can log in. For LangSmith, run a trivial traced call and execute langsmith trace to see the project populate.

Step 3: Instrument one real code path

Wrap your existing OpenAI-compatible client call. Below is a minimal Langfuse trace using the Python SDK. The point is to see whether the platform captures input, output, and token usage without you writing a custom serializer.

from langfuse import Langfuse
from openai import OpenAI

langfuse = Langfuse()
client = OpenAI()

@langfuse.trace(name="chat-completion")
def ask(prompt: str):
    generation = langfuse.generation(name="gpt-call", model="gpt-4o-mini")
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    generation.end(
        output=resp.choices[0].message.content,
        usage={"input": resp.usage.prompt_tokens, "output": resp.usage.completion_tokens},
    )
    return resp.choices[0].message.content

ask("What is the fallback strategy for rate limits?")
langfuse.flush()

If you route through a gateway like n4n.ai, which exposes one OpenAI-compatible endpoint across 240+ models with automatic fallback when a provider is degraded, the same resp.usage object already carries per-token metering and cache hints—no extra parsing needed to populate cost spans.

For LangSmith, the equivalent is a decorator:

from langsmith import traceable
from openai import OpenAI

client = OpenAI()

@traceable(name="chat-completion")
def ask(prompt: str):
    return client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )

Verify

Execute the script. In Langfuse, open the trace and confirm you see a generation span with input/output and token usage. In LangSmith, the trace appears in the project. If you had to fork the client library to get this, that’s a red flag.

Step 4: Test evaluation and dataset workflows

Observability without evaluation is just expensive logging. Create a small dataset of 20 inputs and score outputs. You want to know if the platform treats evaluation as a first-class citizen or a bolt-on.

Langfuse datasets:

dataset = langfuse.create_dataset(name="fallback-q")
for item in [{"input": "Rate limit handling?"}, {"input": "Cache control?"}]:
    langfuse.create_dataset_item(dataset_name="fallback-q", input=item["input"])

# later, run eval
item = langfuse.get_dataset_item(item_id)
out = ask(item.input)
langfuse.score(trace_id=out.trace_id, name="correct", value=1.0)

LangSmith datasets use the client:

from langsmith import Client
client = Client()
dataset = client.create_dataset("fallback-q")
client.create_examples(inputs=[{"q": "Rate limit?"}], dataset_id=dataset.id)

Write a custom evaluator that checks for a keyword in the output. The platform should let you run that evaluator over the dataset and store the score inline with the trace.

Verify

Confirm you can attach a score to a trace and filter traces by score in the UI. If the platform makes scoring a separate product tier, note that in your evaluation.

Step 5: Check cost and latency breakdowns

Your platform should show token counts per span and let you group by model. Pull a trace and inspect the usage object. If you use multiple providers, ensure the UI distinguishes them and reports cached token savings via prompt_tokens_details.cached_tokens when the provider supports it.

A quick check: send one call to a cheap model and one to an expensive model, then confirm the cost view separates them.

def ask_with_model(prompt: str, model: str):
    generation = langfuse.generation(name="call", model=model)
    resp = client.chat.completions.create(model=model, messages=[{"role":"user","content":prompt}])
    generation.end(output=resp.choices[0].message.content, usage={"input": resp.usage.prompt_tokens, "output": resp.usage.completion_tokens})
    return resp

for model in ["gpt-4o-mini", "gpt-4o"]:
    ask_with_model("Explain fallback", model)

Verify

Open the analytics page. You should see two model rows with distinct token totals and a computed cost (even if you supply the price table manually). Latency percentiles should be queryable by model and day.

Step 6: Simulate provider degradation

If your architecture relies on failover, your observability must surface which provider served the token. Using a gateway that honors client routing directives simplifies this, but your trace should still make the fallback explicit.

Write a test that forces a rate limit (e.g., mock 429) and confirm the fallback path logs a distinct span.

@langfuse.trace(name="failover-test")
def test_fallback():
    try:
        client.chat.completions.create(model="broken-provider", messages=[])
    except Exception:
        generation = langfuse.generation(name="fallback", model="gpt-4o-mini")
        generation.end(output="recovered")

Run this in CI weekly so you know the observability path itself doesn’t break during real incidents.

Verify

The trace shows the error span and the fallback generation. If you can’t tell which provider answered, the platform fails your requirement.

Step 7: Decide and document

After the spike, rank candidates on: span fidelity, eval ergonomics, cost visibility, self-host cost, and lock-in. Write a two-page internal note with screenshots from Step 3 and Step 4.

A simple scoring sheet:

| Criterion          | Langfuse | LangSmith |
|--------------------|----------|-----------|
| Self-hostable      | Yes      | No        |
| Tool span nesting  | Yes      | Yes       |
| Score filtering    | Yes      | Yes       |
| Raw export API     | Yes      | Yes       |

Choosing an LLM observability platform becomes a matter of evidence, not slides. Pick the one that let you instrument a second service in under 30 minutes using the same pattern.

Verify success

Your team can stand up the chosen platform, instrument a new service in under 30 minutes using the pattern from Step 3, and query traces by score. That’s the bar.

What to avoid

Pretty dashboards that hide the raw request/response payload are a trap. If you can’t export traces to your own data lake via API, you’re renting your debugging history.

Also skip platforms that require you to change your model client to a proprietary SDK just for logging. OpenAI-compatible interception is table stakes in 2025. If the integration breaks when you add a response_format or a tool call, it’s not ready.

Closing recommendation

Run the seven steps with Langfuse and LangSmith as the two candidates; they cover the spectrum from self-hosted to managed. You will learn more about your own call patterns in a week of instrumentation than in a month of vendor calls.