How Helicone's proxy architecture captures LLM traffic

Helicone proxy architecture sits between your application and the LLM provider, intercepting OpenAI-compatible requests without requiring code changes. That single design decision determines most of its strengths and its limits: you get universal capture across languages, but you also inherit a network hop and a dependency on protocol compatibility. If you understand exactly where the proxy terminates and what it copies, you can decide whether it belongs in your stack.

The core idea: a transparent reverse proxy

Helicone exposes an endpoint that mirrors the OpenAI HTTP API. You change your client’s base_url from https://api.openai.com/v1 to https://api.helicone.ai/v1 and keep the same request shape. The proxy authenticates to Helicone via a header, then forwards the call to the real provider using your provider key. It does not transform the body beyond what routing requires.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.helicone.ai/v1",
    api_key="sk-helicone-...",  # Helicone key
    default_headers={"Helicone-Auth": "Bearer sk-helicone-..."}
)

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain proxies"}]
)

The proxy reads the JSON body, opens an upstream connection to OpenAI (or another mapped provider), and streams the response back. Because it speaks the same HTTP contract, your existing SDK, retries, and type definitions work unchanged. Under the hood it is a stateless middlebox: it terminates TLS, logs metadata, and relays bytes.

What actually gets captured

The Helicone proxy architecture logs the full request path: model, temperature, max_tokens, messages, response token counts, latency buckets, and HTTP status. It also captures custom properties you pass via headers:

curl https://api.helicone.ai/v1/chat/completions \
  -H "Helicone-Auth: Bearer $HELICONE_KEY" \
  -H "OpenAI-API-Key: $OPENAI_KEY" \
  -H "Helicone-Property-App: support-bot" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"hi"}]}'

In the dashboard you see per-request traces with those properties indexed. Streaming responses are captured chunk-by-chunk; the proxy does not buffer unless you enable response logging, which matters for large outputs. Error responses—429s, 5xxs—are recorded with the upstream body so you can debug rate limits without reproducing locally.

What it cannot capture is anything that does not go through the proxy. If a developer hardcodes api.openai.com in a script, or uses a provider-native SDK that bypasses the OpenAI shape, that traffic is invisible. This is the fundamental boundary of any proxy-based observability: coverage equals compliance with the base-URL switch.

Why proxying wins on adoption

The alternative is SDK instrumentation: you wrap every LLM call in a middleware that sends telemetry to Helicone. That gives fine-grained control but requires code changes in every service and language runtime.

import { HeliconeOpenAI } from "@helicone/helicone";
const helicone = new HeliconeOpenAI({ apiKey: HELICONE_KEY });
// must use helicone.chat.completions instead of openai

The Helicone proxy architecture avoids that. A single environment variable (OPENAI_BASE_URL) flips an entire fleet to monitored traffic. For a polyglot shop running Python, Node, and Go, the proxy is the only approach that needs no per-language library. Organizational friction is near zero: platform team sets the variable in the shared deployment manifest, and product teams never touch their code.

The hidden costs of the proxy hop

Adding a reverse proxy is not free.

Latency. Every token now routes through an extra TLS termination and upstream connection. The added median is typically small, but under proxy congestion it can spike. For latency-sensitive agent loops making dozens of sequential calls, that compounds into noticeable user-facing delay.

Availability. Helicone becomes a mandatory middlebox. If their proxy degrades, your LLM calls fail unless you have a fallback base-URL switch. Unlike a local SDK wrapper that can fail open (by skipping telemetry), the proxy is a hard dependency. You should design a health-check that flips base_url back to the provider if Helicone is unreachable for more than a threshold.

Data exposure. Prompts and completions traverse Helicone’s servers. For most teams that is acceptable under their processing terms, but regulated workloads may need self-hosting (Helicone offers a Docker image) or the SDK path where you control the egress.

Protocol drift. When OpenAI adds a new field like parallel_tool_calls, the proxy must forward it untouched. Helicone generally does, but obscure provider extensions (Azure deployment IDs, Anthropic system prompts via non-OpenAI headers) require explicit configuration. You lose the ability to use provider-native SDKs that deviate from the OpenAI shape without a shim.

Auth and routing mechanics

The proxy separates two credentials: the Helicone key (Helicone-Auth) and the provider key (OpenAI-API-Key or similar). This lets you issue scoped Helicone keys to teams while keeping the provider key server-side. Routing is determined by the path and optional headers. To hit Anthropic through the OpenAI-compatible shim, you set the model to anthropic/claude-3-5-sonnet and pass an Anthropic key header.

{
  "Helicone-Auth": "Bearer sk-helicone-xxx",
  "Anthropic-API-Key": "sk-ant-xxx"
}

This is convenient, but it means your proxy configuration now encodes provider topology. A typo in a header silently routes to the wrong model or fails auth upstream. The Helicone proxy architecture does not validate provider keys locally; it relays and observes the error.

How it compares to a full inference gateway

A proxy focused on observability stops at logging and forwarding. An inference gateway like n4n.ai takes the same OpenAI-compatible intercept and adds automatic fallback when a provider is rate-limited, per-token metering, and honoring client routing directives. If you need those control-plane features, Helicone alone will not replace a gateway; you would run both, or choose a gateway with built-in observability.

That said, Helicone’s narrow scope is a strength. It does not try to manage your keys or load balance; it watches. That makes it easier to reason about: the proxy is stateless aside from log shipping, and a single bad deploy on their side cannot silently change your model behavior.

Tradeoff summary

Decisive takeaway

If your goal is to get LLM observability across a messy microservice estate this week, the Helicone proxy architecture is the highest-leverage choice: point the base URL, ship, and watch. Accept the latency and dependency as the price of universal capture. If you operate in a regulated environment or need provider-native features that break the OpenAI shape, invest in the SDK wrapper or a self-hosted proxy. For everyone else, the proxy is the right default, and you can always layer a gateway behind it later without rewriting application code.