Building a multi-tool Haystack agent pipeline

This multi-tool haystack agent pipeline tutorial walks through assembling a Haystack 2.0 Agent that orchestrates several tools inside a declarative Pipeline. You’ll end up with a reproducible setup that routes a user query through a calculator, a mock weather lookup, and a custom function without hand-wiring control flow.

Step 1: Install Haystack 2.0 and dependencies

Start with a clean virtual environment, then install the framework and the OpenAI generator backend. Haystack 2.x ships as haystack-ai on PyPI.

python -m venv .venv && source .venv/bin/activate
pip install haystack-ai openai

Verify the import surface before writing code:

from haystack.components.agents import Agent
from haystack.tools import tool
from haystack.components.generators.chat import OpenAIChatGenerator

If those imports resolve, you are on a version that supports the @tool decorator and the Agent component (Haystack ≥ 2.2).

Step 2: Define your tools as typed functions

The Agent reasons over tool schemas, so every tool must have a precise signature and docstring. The @tool decorator extracts the schema from type hints and the docstring description.

from haystack.tools import tool

@tool
def multiply(a: int, b: int) -> int:
    """Multiply two integers and return the product."""
    return a * b

@tool
def get_weather(city: str) -> str:
    """Return a short weather report for a given city name."""
    # Mock implementation; swap with a real API call in production.
    return f"Sunny in {city}, 22C"

@tool
def slugify(text: str) -> str:
    """Convert a string to a lowercase slug with hyphens."""
    return "-".join(text.lower().split())

Keep tool schemas tight

The LLM sees the function name, argument types, and the first line of the docstring. Avoid vague descriptions—“do stuff” forces the model to guess. If a tool can fail (network, parsing), raise a ValueError inside the function; the Agent will surface the error as an observation and can retry or abort.

Step 3: Point the chat generator at a model endpoint

The Agent needs a chat-capable LLM. Use OpenAIChatGenerator with an explicit model name. If you’d rather not manage multiple provider keys, point the generator at n4n.ai’s OpenAI-compatible endpoint—it fronts 240+ models and applies automatic fallback when a provider is rate-limited or degraded.

from haystack.components.generators.chat import OpenAIChatGenerator

llm = OpenAIChatGenerator(
    model="gpt-4o-mini",
    api_key="your-api-key",
    api_base_url="https://api.n4n.ai/v1",  # optional: any OpenAI-compatible base
    generation_kwargs={"temperature": 0.1}
)

Keep temperature low for tool-calling workloads. High randomness produces malformed arguments and burns iterations.

Step 4: Instantiate the Agent with tools and system prompt

The Agent component wraps the LLM and the tool list. Set max_iterations to bound the reasoning loop—unbounded agents hang on ambiguous prompts.

from haystack.components.agents import Agent

agent = Agent(
    chat_generator=llm,
    tools=[multiply, get_weather, slugify],
    system_prompt="You are a terse assistant. Use the provided tools to answer. "
                  "Call only one tool at a time and stop when the query is resolved.",
    max_iterations=5
)

Exit conditions matter

Haystack’s Agent stops when the model returns a message with no tool calls, or when max_iterations is hit. If you see truncated answers, raise the limit, but first check your tool docstrings—most premature exits come from the model misunderstanding a parameter.

Step 5: Wrap the agent in a Pipeline

A single Agent can be run standalone, but putting it in a Pipeline lets you pre-process inputs and post-process replies. Here we add a PromptBuilder to inject a timestamp, then the agent, then a small function component to strip whitespace.

from haystack import Pipeline, component
from haystack.components.builders import PromptBuilder

@component
class TrimWhitespace:
    @component.output_types(reply=str)
    def run(self, text: str):
        return {"reply": text.strip()}

pipe = Pipeline()
pipe.add_component("builder", PromptBuilder(template="Query at {{ ts }}: {{ q }}"))
pipe.add_component("agent", agent)
pipe.add_component("trim", TrimWhitespace())

pipe.connect("builder.prompt", "agent.prompt")
pipe.connect("agent.replies", "trim.text")

The builder turns a dict {"ts": ..., "q": ...} into a single prompt string. The agent consumes it, runs tools, and emits a list of reply strings. We take the first reply and trim it.

This completes the core of the multi-tool haystack agent pipeline tutorial. The pipeline is now a single callable object you can drop into a FastAPI route or a CLI.

Step 6: Execute and verify the multi-tool flow

Run the pipeline with a query that forces at least two tool calls:

result = pipe.run({
    "builder": {"ts": "2024-06-01T12:00:00Z", "q": "What is 12 * 13? Also, weather in Oslo?"}
})

print(result["trim"]["reply"])

What success looks like

A correct run prints a single sentence that includes 156 and Sunny in Oslo, 22C (or your mock string). To assert programmatically:

reply = result["trim"]["reply"]
assert "156" in reply
assert "Oslo" in reply

If the assertion fails, enable debug logging to see the tool calls:

import logging
logging.basicConfig(level=logging.DEBUG)

Look for tool_calls in the agent’s intermediate state. Common failure: the model calls multiply with string arguments "12" and "13". Haystack’s @tool coercion usually handles it, but if your real tool expects int, tighten the prompt or add a parsing guard.

Operational notes for production

The pattern above is minimal but production-ready if you respect three constraints.

Observability. Wrap each tool with a decorator that logs arguments and latency. The Agent loop is opaque; you need per-call traces to debug why it picked the wrong function.

Fallback. When using a single OpenAI-compatible gateway, provider outages become transparent only if the gateway forwards cache-control hints and honors routing directives. Test by forcing a bad model name and confirming the gateway returns a structured error rather than a hang.

Token metering. Tool schemas and system prompts are sent on every iteration. In a 5-iteration agent, a 200-token prompt becomes 1,000 tokens of input before any output. Use per-token usage metering at the gateway level to catch runaway loops early.

The multi-tool Haystack agent pipeline tutorial you just followed yields a declarative graph: builder → agent → trim. Swap the mock tools for a SQL runner or a vector retriever and the pipeline structure stays identical. That isolation is why Haystack 2.0 beats hand-rolled ReAct loops for maintenance.