Debugging haystack agent pipelines tracing starts with accepting that agents are non-deterministic by design. A single user request can spawn dozens of tool calls, branch across conditional logic, and retry on failure — all before returning an answer. Without structured visibility, you’re guessing. This guide walks through instrumenting Haystack 2.0 pipelines, reading the resulting traces, and using them to fix real problems.
Enable tracing at the pipeline level
Haystack 2.0 ships with OpenTelemetry integration. The fastest path to useful traces is wiring the pipeline’s tracer at construction time, not sprinkling decorators after the fact.
# tracing_setup.py
from haystack import Pipeline
from haystack.tracing import tracer
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME
# Configure once at process startup
provider = TracerProvider(
resource=Resource.create({SERVICE_NAME: "haystack-agent"})
)
provider.add_span_processor(
BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
)
tracer.provider = provider
# Now build your pipeline — all components inherit the tracer
pipeline = Pipeline()
pipeline.add_component("retriever", retriever)
pipeline.add_component("generator", generator)
pipeline.add_component("agent", agent)If you’re running in a managed environment (Kubernetes, Cloud Run, etc.), replace the OTLP endpoint with your collector. The key point: configure the provider before instantiating components. Haystack components read tracer.provider at import time for some internals, and at runtime for span creation.
Understand what Haystack emits by default
Every component that inherits from Component (which is all of them) emits a span per run() invocation. The span name follows the pattern haystack.{component_type}.{component_name}. Attributes include:
haystack.component.input— JSON-serialized inputshaystack.component.output— JSON-serialized outputshaystack.component.duration_ms— wall-clock timehaystack.component.error— exception message if the run failed
Agents add a layer on top. The Agent component emits a parent span for the entire reasoning loop, with child spans for each ToolInvocation. A typical trace hierarchy looks like:
haystack.pipeline.run
└── haystack.agent.agent
├── haystack.tool_invocation.search_web
│ └── haystack.component.run (WebSearch)
├── haystack.tool_invocation.read_url
│ └── haystack.component.run (UrlFetcher)
└── haystack.component.run (Generator)This hierarchy is your map. When latency spikes or the agent hallucinates a tool call, you trace the path from the root span down to the offending leaf.
Correlate traces with logs and metrics
Spans alone don’t tell you why a tool failed. You need the log line that happened inside the tool’s run() method. Haystack doesn’t auto-correlate logs to traces — you must inject the trace context.
# components/web_search.py
from haystack import component
from opentelemetry.trace import get_current_span, SpanContext
import logging
import json
logger = logging.getLogger(__name__)
@component
class WebSearch:
@component.output_types(results=list)
def run(self, query: str, top_k: int = 5):
span = get_current_span()
trace_id = span.get_span_context().trace_id if span.get_span_context() else None
# Structured log with trace_id for correlation
logger.info(json.dumps({
"event": "web_search_start",
"query": query,
"top_k": top_k,
"trace_id": f"{trace_id:032x}" if trace_id else None
}))
try:
results = self._search(query, top_k)
logger.info(json.dumps({
"event": "web_search_complete",
"result_count": len(results),
"trace_id": f"{trace_id:032x}" if trace_id else None
}))
return {"results": results}
except Exception as e:
logger.error(json.dumps({
"event": "web_search_error",
"error": str(e),
"trace_id": f"{trace_id:032x}" if trace_id else None
}), exc_info=True)
raiseNow a log query for trace_id:abc123 in your aggregator (Datadog, Loki, CloudWatch) surfaces every log line from that pipeline execution, not just the component boundaries.
Spot the three most common agent failure patterns
1. Tool selection loops
The agent calls the same tool repeatedly with slightly different parameters, never converging. In the trace, you’ll see a deep chain of haystack.tool_invocation.* spans with near-identical inputs.
Fix: Add a max_tool_invocations limit to the agent and surface the limit in the trace as an attribute.
from haystack.agents import Agent
agent = Agent(
tools=tools,
max_tool_invocations=5, # hard stop
system_prompt="...",
)
# In your tracing setup, add a custom span attribute when the limit is hit
from opentelemetry.trace import get_current_span
def _wrap_agent_run(agent):
original_run = agent.run
def wrapped_run(*args, **kwargs):
span = get_current_span()
try:
return original_run(*args, **kwargs)
except Exception as e:
if "max_tool_invocations" in str(e):
span.set_attribute("haystack.agent.max_invocations_exceeded", True)
raise
return wrapped_run
agent.run = _wrap_agent_run(agent)2. Silent tool failures
A tool returns an error structure instead of raising, and the agent treats it as valid output. The trace shows haystack.component.error as null, but the output contains {"error": "rate limited"}.
Fix: Enforce a contract — tools must raise on failure. Validate in a pipeline hook.
from haystack import Pipeline, component
from haystack.core.pipeline.hooks import PipelineHook
class ToolOutputValidator(PipelineHook):
def after_component_run(self, component_name, component, inputs, outputs):
for output_name, output_value in outputs.items():
if isinstance(output_value, dict) and "error" in output_value:
raise RuntimeError(
f"Component {component_name} returned error in output: {output_value['error']}"
)
pipeline = Pipeline()
pipeline.add_hook(ToolOutputValidator())Now the failure appears in the trace as haystack.component.error, and your alerting catches it.
3. Context window exhaustion
The agent’s conversation history grows until the generator fails with a token limit error. The trace shows a single haystack.component.run span for the generator with a 400 error, but the root cause is 15 tool invocations upstream.
Fix: Emit a running token count as a span attribute on each tool invocation.
from haystack.agents import Tool
class TokenCountingTool(Tool):
def invoke(self, *args, **kwargs):
result = super().invoke(*args, **kwargs)
span = get_current_span()
# Rough estimate: 1 token ≈ 4 chars for English
token_estimate = len(str(result)) // 4
span.set_attribute("haystack.tool.output_token_estimate", token_estimate)
return resultPair this with a dashboard that sums haystack.tool.output_token_estimate per trace. When the sum approaches your model’s context window, you’ve found the budget leak.
Add custom spans for business logic
Haystack’s automatic spans cover component boundaries. They don’t cover your decision points — routing logic, fallback selection, cache hits. Add those manually.
# services/routing.py
from opentelemetry.trace import get_tracer
from haystack import component
tracer = get_tracer("routing")
@component
class ModelRouter:
@component.output_types(model=str, provider=str)
def run(self, query: str, complexity: str):
with tracer.start_as_current_span("model_routing_decision") as span:
span.set_attribute("routing.query_length", len(query))
span.set_attribute("routing.complexity_hint", complexity)
if complexity == "high":
decision = {"model": "gpt-4o", "provider": "openai"}
elif complexity == "low":
decision = {"model": "llama-3.1-8b", "provider": "together"}
else:
decision = {"model": "claude-3-haiku", "provider": "anthropic"}
span.set_attribute("routing.selected_model", decision["model"])
span.set_attribute("routing.selected_provider", decision["provider"])
return decisionThis span appears in the trace between the agent’s reasoning and the generator call, letting you audit routing decisions without parsing logs.
Handle streaming responses
Streaming generators emit one span for the entire stream, not per chunk. If you need per-chunk visibility (for latency percentiles, token-level errors), wrap the generator.
# components/streaming_generator.py
from haystack import component
from haystack.dataclasses import StreamingChunk
from opentelemetry.trace import get_tracer
tracer = get_tracer("streaming")
@component
class TracedGenerator:
def __init__(self, generator):
self.generator = generator
@component.output_types(replies=list, meta=list)
def run(self, prompt: str, streaming_callback=None):
with tracer.start_as_current_span("traced_generation") as span:
span.set_attribute("generation.prompt_tokens", len(prompt) // 4)
def traced_callback(chunk: StreamingChunk):
span.add_event("generation_chunk", {
"token": chunk.content,
"chunk_index": chunk.meta.get("index", 0)
})
if streaming_callback:
streaming_callback(chunk)
result = self.generator.run(prompt, streaming_callback=traced_callback)
total_tokens = sum(len(r) for r in result["replies"]) // 4
span.set_attribute("generation.completion_tokens", total_tokens)
return resultThe add_event calls create timestamped events on the span. Most tracing backends render these as a timeline within the span — useful for spotting stalls mid-stream.
Pitfalls and tradeoffs
Sampling: At volume, you cannot trace 100% of requests. Use tail-based sampling — keep traces with errors, high latency, or specific attributes (e.g., haystack.agent.max_invocations_exceeded). Head-based sampling loses the very traces you need to debug.
Cardinality: Avoid high-cardinality attributes like query or user_id on spans. They explode your tracing backend’s index. Put them on logs instead, correlated by trace_id.
Serialization cost: Haystack serializes inputs/outputs to JSON for span attributes. Large documents (e.g., 50 retrieved passages) blow up span size and ingestion cost. Truncate in a custom component wrapper:
def _truncate_for_trace(data, max_chars=2000):
s = json.dumps(data, default=str)
return s[:max_chars] + ("..." if len(s) > max_chars else "")
class TracedComponent:
def __init__(self, component):
self.component = component
def run(self, **kwargs):
span = get_current_span()
span.set_attribute("haystack.component.input", _truncate_for_trace(kwargs))
result = self.component.run(**kwargs)
span.set_attribute("haystack.component.output", _truncate_for_trace(result))
return resultClock skew: In distributed setups, component spans may show negative duration if the collector’s clock differs from the worker’s. Use NTP everywhere, or rely on the collector’s received timestamp for ordering.
Operationalize: from trace to fix
- Alert on error rate per component:
haystack.component.errorexists → page on-call. - Dashboard p95 latency per tool:
haystack.tool_invocation.*duration → catch regressions when a provider degrades. - Weekly trace review: Sample 10 failed traces, 10 slow traces. Categorize root causes. Feed back into prompt engineering, tool design, or routing logic.
The tracing setup is not a one-time task. As you add tools, change models, or modify the agent’s system prompt, the failure modes shift. Treat your tracing instrumentation as part of the pipeline’s contract — version it, test it, and review it like code.