CrewAI multi-agent setups with gateway-routed models

CrewAI multi-agent gateway-routed models let you run a crew of specialized agents across multiple underlying providers without hardcoding vendor SDKs. By pointing CrewAI’s LLM clients at an OpenAI-compatible gateway, you centralize model selection, failover, and cost metering behind one endpoint. This guide walks through a production-grade setup you can run end to end today.

Step 1: Choose and configure your gateway endpoint

Pick a gateway that exposes an OpenAI-compatible /v1/chat/completions surface. A gateway such as n4n.ai fronts 240+ models behind one URL and handles automatic fallback when a provider is rate-limited or degraded. You only need two secrets: the base URL and an API key.

export GATEWAY_BASE_URL="https://api.n4n.ai/v1"
export GATEWAY_API_KEY="sk-your-gateway-key"

If you run your own gateway (LiteLLM, OpenRouter, or similar), swap the URL. The rest of the code is identical because CrewAI talks to any LangChain chat model that implements the OpenAI signature.

Store these in a .env file in real deployments. Never bake them into source.

Step 2: Install and pin dependencies

CrewAI moves fast. Pin versions to avoid silent breaking changes in the Agent or LLM interfaces.

pip install "crewai==0.28.0" "langchain-openai==0.1.7" "python-dotenv==1.0.1"

We use langchain-openai because CrewAI accepts any LangChain chat model as an llm argument. This gives us direct control over base_url and api_key without waiting for CrewAI to add native support for a specific gateway. Load env vars early:

from dotenv import load_dotenv
load_dotenv()

Step 3: Define a gateway-routed LLM factory

Write a small factory that returns a configured ChatOpenAI instance. The model parameter is just a string; the gateway interprets routing hints. For example, gpt-4o may route to OpenAI, while anthropic/claude-3-5-sonnet routes to Anthropic if the gateway supports that namespace.

import os
from langchain_openai import ChatOpenAI

def gateway_llm(model: str, temperature: float = 0.0) -> ChatOpenAI:
    return ChatOpenAI(
        model=model,
        temperature=temperature,
        base_url=os.environ["GATEWAY_BASE_URL"],
        api_key=os.environ["GATEWAY_API_KEY"],
        max_tokens=2000,
        timeout=60,
        max_retries=3,
        streaming=False,
    )

The max_retries here is client-side. A good gateway adds its own upstream failover, so you don’t need aggressive retry loops. Set streaming=True only if you plan to consume token deltas in a custom UI; CrewAI’s default verbose logging works fine with buffered responses.

Keep the factory in a llm.py module. Centralizing construction prevents model-string drift across agents.

Step 4: Build agents with distinct model assignments

The power of CrewAI multi-agent gateway-routed models is per-agent model selection. A planner can use a frontier model; a summarizer can use a cheap one. All traffic goes through the same gateway, so you get unified logs and one billing relationship.

from crewai import Agent

planner = Agent(
    role="Research Planner",
    goal="Break the query into a step-by-step plan",
    backstory="Senior analyst who thinks in outlines",
    llm=gateway_llm("gpt-4o", temperature=0.2),
    verbose=True,
)

executor = Agent(
    role="Web Researcher",
    goal="Execute the plan and gather facts",
    backstory="Fast junior researcher",
    llm=gateway_llm("anthropic/claude-3-haiku", temperature=0.0),
    verbose=True,
)

writer = Agent(
    role="Draft Writer",
    goal="Synthesize findings into a report",
    backstory="Precise technical writer",
    llm=gateway_llm("mistral/mixtral-8x7b-instruct", temperature=0.3),
    verbose=True,
)

If your gateway does not recognize a namespace, it will either fallback or error. Test model strings with a single call before wiring them into a crew:

print(gateway_llm("anthropic/claude-3-haiku").invoke("ping").content)

Cost discipline is the real win. You don’t need a $20/M-token model to format JSON when a $0.25/M-token model does it reliably.

Step 5: Define tasks and crew process

Tasks reference the agent that owns them. Use expected_output to force structured handoffs between agents. CrewAI passes prior task output into context automatically.

from crewai import Task, Crew, Process

plan_task = Task(
    description="Create a 3-step research plan for: {topic}",
    expected_output="Numbered list of research steps",
    agent=planner,
)

research_task = Task(
    description="Run the plan and collect sources",
    expected_output="Bullet list of facts with URLs",
    agent=executor,
    context=[plan_task],
)

write_task = Task(
    description="Write a 200-word briefing from the research",
    expected_output="Markdown briefing",
    agent=writer,
    context=[research_task],
)

crew = Crew(
    agents=[planner, executor, writer],
    tasks=[plan_task, research_task, write_task],
    process=Process.sequential,
    verbose=True,
)

For parallel fan-out, switch to Process.hierarchical and give the manager agent a strong model. Sequential is simpler to debug; use it first.

Step 6: Run the crew and observe routing

Kick off the crew with a topic. CrewAI streams agent logs to stdout when verbose=True.

result = crew.kickoff(inputs={"topic": "edge caching for LLM gateways"})
print(result)

You should see each agent invoke its assigned model through the gateway URL. If you watch gateway access logs, you’ll notice the same source IP and API key, but different model fields in the request bodies. That’s the clearest signal that CrewAI multi-agent gateway-routed models are working: one egress point, many backends.

Step 7: Verify success and inspect metering

Success means three things: the crew returns a coherent briefing, each agent used its designated model, and you can account for tokens spent.

To confirm model routing, enable debug on the LLM:

import logging
logging.basicConfig(level=logging.DEBUG)

The HTTP request logs show the POST to your GATEWAY_BASE_URL with the correct model. For token accounting, gateways with per-token usage metering expose totals per request. n4n.ai, for instance, writes per-token usage to its metering logs so you can attribute cost to the planner vs. the writer. Even without that, the OpenAI-compatible response includes a usage object you can capture by subclassing or by reading the gateway’s dashboard.

If a provider is degraded, the gateway should fallback automatically. You can simulate this by temporarily routing an agent to a nonexistent model ID; a well-built gateway returns a clear error or substitutes a configured fallback rather than hanging your crew.

Step 8: Advanced — cache-control and hierarchical routing

Production crews repeat context. Forward provider cache-control hints to cut latency and cost. The gateway forwards these headers if you pass them through the LangChain client.

cached_llm = ChatOpenAI(
    model="gpt-4o",
    base_url=os.environ["GATEWAY_BASE_URL"],
    api_key=os.environ["GATEWAY_API_KEY"],
    extra_headers={"x-cache-control": "ephemeral"},
)

In a hierarchical crew, the manager model should be the most capable one because it parses worker outputs and decides completion.

manager = Agent(
    role="Crew Manager",
    goal="Coordinate workers and finalize",
    llm=gateway_llm("gpt-4o"),
    verbose=True,
)

crew = Crew(
    agents=[planner, executor, writer],
    tasks=[plan_task, research_task, write_task],
    process=Process.hierarchical,
    manager_agent=manager,
)

CrewAI multi-agent gateway-routed models keep this manageable: you change one factory, not ten SDK integrations. When a new model drops, you add one string to the gateway’s routing table and point an agent at it.

Common pitfalls

Wrong base_url path. Some gateways require /v1 at the end, some don’t. Match exactly what the docs say or you’ll get 404s on every call.

Model string drift. Provider model names change. Centralize them in the factory as constants like PLANNER_MODEL = "gpt-4o".

Timeout storms. Default LangChain timeout is 60s. Agents that wait on each other can exceed it. Raise timeout to 120 or use async execution via crew.kickoff_async().

Verbose noise. verbose=True is great for dev, terrible for prod logs. Gate it behind an env flag: verbose=os.getenv("CREW_VERBOSE") == "1".

Ignoring fallback semantics. Know whether your gateway throws or substitutes on provider error. Code your error handling accordingly; don’t assume the crew will self-heal.

What you get

You now have a CrewAI crew where each agent picks its own model, all requests flow through one gateway, and you can meter or reroute without touching agent code. That’s the core value of CrewAI multi-agent gateway-routed models: vendor lock-in stays at the gateway, not in your agent logic.

If you need to add a new provider, you update the gateway’s routing table, not your Python files. That’s how you keep a multi-agent system deployable in a fragmented model landscape without rewriting your framework integration every quarter.