Most LLM agent outages are not crashes; they are slow, expensive, and partially wrong. Effective agent monitoring latency cost errors requires treating those three dimensions as linked signals emitted from every step of an agent run. If you only graph them separately, you will miss the interaction where a retry spikes cost and hides a latency regression.
Unify the three pillars in a single trace
An agent run is a tree: planner → tool call → model re-entry → final answer. Latency, cost, and errors each attach to a node. Your observability stack must correlate them by a single trace ID, or you will spend hours reconstructing what a “slow” request actually did.
OpenTelemetry is the pragmatic default. Span attributes are free-form key-value pairs; use them to stamp cost_usd, error_class, and latency_ms on the same object. Do not push metrics to one system and logs to another without a join key.
1. Instrument every span, not just the HTTP boundary
Wrapping only the outer HTTP handler hides the real culprits: vector lookups, tool authentication, and JSON parsing. Start with a minimal context manager and expand.
import time
from contextlib import contextmanager
@contextmanager
def span(name, trace_id):
start = time.perf_counter()
attrs = {"trace_id": trace_id, "span": name}
try:
yield attrs
finally:
attrs["latency_ms"] = (time.perf_counter() - start) * 1000
# emit to collector
print(attrs)Common pitfall: developers instrument the LLM call but skip the tool execution. A SQL query that takes 800 ms inside a “fast” agent is invisible if you only time client.chat.completions.create.
2. Decompose latency into phases
A single “model latency” number is useless. Break it into prefill (prompt processing), decode (token generation), and external round-trips. Prefill scales with context size; decode scales with output length; tool calls scale with your backend.
import statistics
samples = [120, 450, 300, 2000, 180] # ms per agent run
p50 = statistics.median(samples)
p95 = sorted(samples)[int(0.95 * len(samples)) - 1]Alert on p95, not mean. A mean of 430 ms above hides the 2-second tail that churns users. Tradeoff: high-resolution histograms cost storage. Use exponential bucket histograms and drop raw spans for successful sub-50 ms calls after 24 hours.
3. Attribute cost to the span that spent it
LLM cost is per-token. The OpenAI-compatible response object gives you exactly what you need:
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Summarize"}]
)
u = response.usage
# prompt_tokens, completion_tokens are integers
cost = u.prompt_tokens * 0.00015 + u.completion_tokens * 0.0006When routing through a gateway such as n4n.ai, the same usage object is returned and per-token metering is honored, so you can pipe it straight into span attributes without custom parsing. Do not wait for a monthly invoice to attribute spend—assign cost at span close.
Pitfall: ignoring cached token discounts. If your provider supports prompt caching, the prompt_tokens_details.cached_tokens field changes cost materially. Forward cache-control hints and record the discounted count.
4. Classify errors beyond status codes
A 200 with malformed tool output is an error. Build a taxonomy:
{
"error_type": "schema_drift",
"span": "parse_tool_output",
"detail": "missing field 'order_id'",
"trace_id": "abc123"
}Categories we use in production:
- Provider 4xx/5xx: auth, rate limit, outage.
- Tool timeout: downstream API slow.
- Schema drift: model returned JSON that fails validation.
- Hallucinated action: tool called with invalid params that passed lint but failed business logic.
Treating all non-200 as “provider error” masks the fact that 80% of your failures are schema drift fixable in your prompt.
5. Implement fallback with explicit budgets
Retries are necessary; unbounded retries are bankruptcy. Wrap calls in a loop with a cost ceiling.
max_spend = 0.05 # USD
spent = 0.0
last_err = None
for attempt in range(3):
res = call_model()
spent += cost_of(res)
if res.ok:
break
last_err = res.error
if spent > max_spend:
raise BudgetExceeded(f"spent {spent} > {max_spend}")Gateways like n4n.ai perform automatic fallback when a provider is degraded, but you must still record the redirected attempt as a child span to keep agent monitoring latency cost errors accurate. Otherwise your latency chart shows one call while cost shows two.
Tradeoff: aggressive fallback improves reliability but obscures provider-specific regressions. Tag the resolved provider on the span so you can spot a consistently flaky upstream.
6. Sampling and overhead tradeoffs
Full-fidelity tracing on every agent step in a high-traffic system will saturate your collector. Use head-based sampling for healthy traffic (e.g., 10%) and tail-based sampling to always keep errors and p99 latency outliers.
Pitfall: sampling before cost attribution. If you drop a span, you drop its cost_usd. Compute cost synchronously at call time and emit a lightweight counter even if the detailed span is sampled out.
7. Alert on ratios, not absolutes
Raw latency and cost numbers drift with traffic. Alert on derived signals:
- Cost per successful task (not per request).
- Error rate per tool (not global 5xx).
- Latency per 1k context tokens (normalizes for prompt size).
Example threshold: page if cost per resolved support ticket exceeds 2× trailing 7-day median. This catches silent model upgrades that quietly triple spend.
Pre-flight checklist
- Every agent step emits a span with
trace_id,latency_ms,cost_usd,error_type. - Latency dashboards show p50/p95/p99 per phase, not averages.
- Cost is attributed at token level using
usagefrom the response. - Error taxonomy includes non-HTTP failures (schema, timeout, logic).
- Retry loops carry a hard spend cap and log fallback spans.
- Sampling preserves cost counters even when spans are dropped.
- Alerts fire on ratios tied to business outcomes.
Ship this before your next agent feature, not after the incident.