Five signals to monitor in production AI agents

When you monitor production AI agents, standard APM metrics like CPU and HTTP 500s hide the real failure modes. The five signals below are the ones we treat as mandatory before any agent handles live traffic, because they catch degradation specific to LLM orchestration and multi-step reasoning.

1. Token throughput and latency per model route

An agent rarely calls one model. It may use a cheap model for classification, a flagship model for generation, and an embedding model for retrieval. Aggregate latency across the whole trace masks which route is slow. You need p95 time-to-first-token (TTFT) and tokens/sec broken down by model and endpoint, not just a single “agent response time” line.

If you route through a gateway, capture the model field from the response and log it alongside latency. A single OpenAI-compatible call returns the model actually served, which is critical when a router substitutes a fallback:

{
  "model": "anthropic/claude-3.5-sonnet",
  "usage": { "prompt_tokens": 120, "completion_tokens": 45 },
  "metrics": { "ttft_ms": 320, "tokens_per_sec": 140 }
}

n4n.ai exposes per-token usage metering on an OpenAI-compatible endpoint covering 240+ models, so you can attribute latency to the exact route without wrapping each provider SDK. That removes a class of instrumentation bugs where the wrong model name gets tagged.

Plot p95 TTFT per model weekly. A 2x regression in a smaller model often predicts a downstream timeout in the orchestrator because the agent’s outer loop blocks on it. Alert when a route’s p95 TTFT exceeds a static threshold (e.g., 2s for a 7B class model) or shifts >30% week-over-week. Export these as OpenTelemetry histograms, not just logs.

2. Tool call failure rate and retry loops

Agents invoke tools via function calls. A single failed tool is normal; a loop of identical calls is a stuck agent burning tokens. Monitor the ratio of failed tool calls to total, and track consecutive identical (name + normalized args) invocations as a distinct signal.

Capture traces as structured events so you can reconstruct the loop offline:

{"agent_step": 12, "tool": "sql_query", "args": {"q": "SELECT * FROM users"}, "status": "error", "error": "timeout"}
{"agent_step": 13, "tool": "sql_query", "args": {"q": "SELECT * FROM users"}, "status": "error", "error": "timeout"}
{"agent_step": 14, "tool": "sql_query", "args": {"q": "SELECT * FROM users"}, "status": "error", "error": "timeout"}

Set a hard limit: if the same tool+args repeats three times, kill the run and surface a panic metric. In our stacks, a tool failure rate above 8% on a critical path triggers a page, because it correlates with user-visible stalls. The LLM will often “apologize and retry” rather than switch strategy, so you cannot rely on self-correction.

Instrument the orchestrator to emit a counter agent_tool_failures_total and a gauge agent_max_repeat_loop. Hash the normalized arguments to detect semantic repeats even when whitespace differs. This catches the classic “agent calls weather API with same city 14 times” bug before it drains your quota.

3. Context window utilization and truncation

Middle-tier agents silently truncate history when they hit context limits. If you monitor production AI agents, you must know how close prompts get to the model’s limit and whether your middleware quietly drops messages. A prompt that is 99% of max tokens today may drop a system instruction tomorrow when the user pastes a long doc.

Compute prompt token count before send. A minimal check:

import tiktoken

def count_tokens(text, model="gpt-4o"):
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))

prompt = build_prompt(state)
limit = 128_000
if count_tokens(prompt) > 0.9 * limit:
    metrics.incr("context_near_limit")

Watch the distribution of prompt_tokens / max_context per model. A sudden jump to 0.95+ means summarization or eviction logic kicked in—verify it didn’t drop system instructions. Log an explicit truncation_event whenever you slice the conversation, with the number of tokens removed and which role lost messages.

Provider context limits are not uniform, and some gateways forward cache-control hints that change effective window behavior. Track utilization per model version, because a “128k” model from one provider may count system prompts differently than another.

4. Cost per completed task (including hidden retries)

Per-token cost is easy; per-task cost is the number that matters. Agents retry, fall back, and call multiple models for one user request. Sum all token spend plus tool compute for a single agent goal ID, not per HTTP request.

Attach a task_id to every LLM call and aggregate offline:

# in metrics sink
def observe(task_id, usage, model):
    cost = PRICING[model]["in"] * usage.prompt + PRICING[model]["out"] * usage.completion
    redis.hincrby(f"task:{task_id}:cost", model, cost)

If you use a gateway with automatic fallback, ensure the metering includes those attempted calls. A gateway that honors client routing directives and forwards provider cache-control hints—like n4n.ai—lets you attribute cache hits to cost reduction accurately instead of guessing. A task that costs 10x the median is either a loop bug or a misrouted prompt.

Track task_cost_percentile and alert on p99 spikes. Separate embedding calls from generation; embedding storms during retrieval can dominate cost in RAG agents. Report cost per successful task, excluding aborted runs, so product can reason about unit economics.

5. Semantic drift and output quality regression

Latency and errors can be green while the agent starts answering nonsense. You need a lightweight quality signal: automated eval scores, user thumbs, or heuristic guards (e.g., “answer must contain a valid JSON block”). These catch model updates you didn’t choose.

Implement a post-hoc checker that runs on every completion:

def validate_output(text):
    if not text.strip().startswith("{"):
        return 0.0
    try:
        json.loads(text)
        return 1.0
    except ValueError:
        return 0.5

Feed this into a rolling mean per agent version. When a new deploy drops the mean below baseline by 5%, roll back. Combine with explicit user feedback events if available; even 20 labeled examples per day is enough to spot a regression.

Semantic drift often follows a provider model update you didn’t pin. Pin model versions in production and monitor production AI agents for sudden eval drops when a route switches. Shadow-test new model versions on a fraction of traffic before promoting them.

Summary

These five signals give you enough to run agents without flying blind. Wire them into the same dashboard as your infra metrics, and treat LLM-specific traces as first-class telemetry rather than afterthought logs.