Arize Phoenix for open-source LLM evaluation and tracing

If you’re shipping LLM features without visibility into prompt, context, and token flow, you’re debugging blind. Arize Phoenix open-source tracing gives you OpenTelemetry-based spans for LLM calls, retrievers, and tool use while keeping your data in your own infrastructure. This guide lays out a concrete path from local install to evaluation loops, and calls out the operational tradeoffs you’ll face when self-hosting observability.

1. Stand up Phoenix locally

Phoenix runs as a single process with a built-in UI and an OpenTelemetry collector. The fastest path is pip:

pip install arize-phoenix
python -m phoenix.server.main serve

This starts the UI on http://localhost:6006 and an OTLP endpoint on http://localhost:4317. For containerized workloads:

docker run -p 6006:6006 -p 4317:4317 arizephoenix/phoenix:latest

Keep the collector port separate from the UI. If you later move to a shared host, put TLS in front of both or keep them on a private network. Phoenix stores traces in an embedded SQLite store by default, which is fine for dev but not for multi-day retention.

2. Instrument your LLM client

Arize Phoenix open-source tracing relies on OpenInference instrumentations that wrap popular SDKs. For OpenAI-compatible clients, install the instrumentation package:

pip install openinference-instrumentation-openai

Configure the tracer before you import your LLM code:

from phoenix.otel import register
from openinference.instrumentation.openai import OpenAIInstrumentor

register(
    project_name="my-llm-app",
    endpoint="http://localhost:4317",
)
OpenAIInstrumentor().instrument()

Now any openai or compatible client calls emit spans automatically. If you route through a gateway like n4n.ai that exposes a single OpenAI-compatible endpoint with fallback across 240+ models, point your client’s base_url at the gateway and the spans still capture latency, token counts, and errors without extra code.

Pitfall: instrumentation must be applied before the client is constructed. Lazy imports inside functions will miss the patch. Do it at the top of your entrypoint. Another pitfall: async OpenAI clients require the openai v1+ instrumentation; older shims leak contexts.

You can also drive configuration via environment variables to avoid hard-coding endpoints:

import os
os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = "http://localhost:4317"
os.environ["PHOENIX_PROJECT_NAME"] = "my-llm-app"
from phoenix.otel import register
register()  # picks up env vars

3. Add retrieval and tool spans manually

Auto-instrumentation covers the model call, but RAG pipelines need custom spans for vector lookups and tool execution. Use the OTel API directly:

from opentelemetry import trace
tracer = trace.get_tracer("rag")

with tracer.start_as_current_span("retrieve") as span:
    span.set_attribute("embedding.model", "text-embedding-3-small")
    docs = vector_store.search(query)
    span.set_attribute("retrieval.count", len(docs))
    span.set_attribute("retrieval.latency_ms", 42)

This keeps your trace tree faithful to the actual execution path. Without it, you’ll see a flat LLM span and wonder why latency spikes. For tool calls, set tool.name and tool.input attributes following OpenInference conventions so the UI renders them correctly.

4. Run LLM-as-judge evaluations

Phoenix ships phoenix.evals for offline and online evaluation. A typical correctness check uses a judge model to label outputs:

from phoenix.evals import llm_classify, OPENAI_LLM_AS_JUDGE
from phoenix.evals.templates import PROMPT_TEMPLATE

labels = ["correct", "incorrect"]
df = llm_classify(
    data=spans_to_df,
    template=PROMPT_TEMPLATE,
    model=OPENAI_LLM_AS_JUDGE,
    labels=labels,
    provide_explanation=True,
)

The output is a pandas DataFrame you can join with trace IDs. Store it back as an evaluation:

from phoenix.client import Client
px = Client(endpoint="http://localhost:6006")
px.log_evaluations(df, project_name="my-llm-app")

Tradeoff: LLM judges add latency and cost. Run them asynchronously post-hoc on sampled traces, not inline on every request, unless you have a hard real-time quality gate. Also, judges are not ground truth; calibrate them against human labels quarterly.

A common mistake is evaluating only the final answer. Capture intermediate spans (retrieval, reasoning) and eval those separately to localize failures.

5. Query traces programmatically

The UI is good for ad-hoc debugging, but you’ll want to aggregate. Phoenix exposes a Python client:

from phoenix.client import Client
px = Client()
df = px.get_spans_dataframe(project_name="my-llm-app")
slow = df[df["llm.token_count.total"] > 4000]

Span attributes are semi-structured. Filter on known keys like llm.token_count.total rather than free-text. If you emit custom attributes, document them in code. For complex queries, use the GraphQL endpoint:

query Spans {
  spans(condition: {projectName: "my-llm-app"}) {
    edges { node { name latencyMs attributes } }
  }
}

Pitfall: the default SQLite backend slows down after a few hundred thousand spans. Migrate to Postgres before that.

6. Production tradeoffs of self-hosted tracing

Arize Phoenix open-source tracing puts you on the hook for storage and uptime. Spans accumulate fast; a busy app emits megabytes per hour. Set a retention policy:

• Use the --storage flag with a Postgres backend for durable retention.
• Sample 10–20% of traces in high-volume services via OTel samplers.
• Scrub PII before export if you handle user data.

Another tradeoff: the collector is a single process by default. For multi-node deployments, point all agents at a central OTLP gateway (or a load-balanced Phoenix) to avoid fragmented traces. You also own the upgrade cycle; semantic conventions change between minor versions.

7. Avoid context propagation leaks

OTel context travels via thread-local or async context vars. If you spawn threads or use asyncio without proper propagation, child spans detach. Use opentelemetry-context helpers:

from opentelemetry.context import attach, detach, set_value
ctx = attach(set_value("tenant_id", "acme"))
# do work
detach(ctx)

Missing parent spans make Phoenix show orphaned LLM calls. Always verify the first deploy with a synthetic request and check the trace tree. In FastAPI or Django, use the framework middleware provided by OpenInference to bridge request scopes.

8. Evaluate continuously, not just at launch

Set up a nightly job that replays a golden set of prompts through your pipeline and logs evaluations to Phoenix. Compare runs by version tag:

px.log_evaluations(nightly_df, project_name="my-llm-app", dataset="golden-v2")

This catches regressions when you swap models or change prompts. Arize Phoenix open-source tracing makes the diff visible because every span carries the model name and prompt hash if you set them as attributes:

span.set_attribute("llm.model", "gpt-4o-mini")
span.set_attribute("prompt.hash", "a1b2c3")

Wire this into CI: fail the build if eval score drops beyond a threshold versus the previous tag.

9. When not to use Phoenix

If your compliance posture forbids any local retention of prompt text, you’ll need a redaction proxy ahead of the collector. If your team lacks ops capacity, the hosted Arize service may be cheaper than running Postgres and Phoenix HA. Open-source does not mean zero-cost; it means you control the bill and the bytes. For tiny prototypes, console logging may suffice until you actually need to debug a multi-step agent.

10. Quick reference checklist

1. Install Phoenix and instrument before client init.
2. Add manual spans for retrieval and tools.
3. Run judges offline; log evaluations with trace IDs.
4. Query via client or GraphQL; export to Postgres early.
5. Sample in production; scrub PII.
6. Verify context propagation on first deploy.
7. Nightly eval diffs catch regressions.

Follow that order and Arize Phoenix open-source tracing becomes a debugging asset rather than a backlog item.