CrewAI agents powered by local Qwen2.5 models

Running multi-agent workflows locally eliminates API costs, latency variance, and data egress concerns. Qwen2.5 models — particularly the 7B and 14B instruct variants — hit a sweet spot for agentic reasoning: strong instruction following, solid tool-use capability, and low enough VRAM to run on a single consumer GPU. This guide walks through wiring CrewAI to local Qwen2.5 via Ollama, configuring agents with proper prompts, and verifying the pipeline end to end.

Step 1: Install Ollama and pull Qwen2.5

Ollama provides the simplest local inference server with an OpenAI-compatible API endpoint. Install it for your platform, then pull the model you want. For most agent workloads, start with qwen2.5:7b-instruct (4.7 GB) or qwen2.5:14b-instruct (9 GB) if you have 16 GB+ VRAM.

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows: download from ollama.com/download

Start the daemon and pull the model:

ollama serve &
ollama pull qwen2.5:7b-instruct

Verify the model responds:

curl -s http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen2.5:7b-instruct","messages":[{"role":"user","content":"Reply with only: pong"}],"stream":false}' \
  | jq -r '.message.content'
# Expected: pong

If you hit memory pressure, quantize further: ollama pull qwen2.5:7b-instruct-q4_K_M (3.9 GB). The Q4_K_M quantization retains most reasoning quality for agent tasks.

Step 2: Create a Python environment and install CrewAI

Use a virtual environment to isolate dependencies. CrewAI 0.28+ supports arbitrary OpenAI-compatible endpoints via the base_url parameter on ChatOpenAI.

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install "crewai[tools]" langchain-openai

The langchain-openai package provides the ChatOpenAI class that CrewAI uses under the hood. Pin versions if you need reproducibility:

# requirements.txt
crewai==0.30.1
langchain-openai==0.1.25
ollama==0.2.1

Step 3: Configure the local LLM wrapper

Create a small module that returns a configured ChatOpenAI instance pointing at Ollama. This keeps your agent definitions clean and makes swapping models trivial.

# llm.py
from langchain_openai import ChatOpenAI

def get_local_llm(
    model: str = "qwen2.5:7b-instruct",
    base_url: str = "http://localhost:11434/v1",
    temperature: float = 0.1,
    max_tokens: int = 4096,
) -> ChatOpenAI:
    """
    Returns a ChatOpenAI client wired to Ollama's OpenAI-compatible endpoint.
    """
    return ChatOpenAI(
        model=model,
        base_url=base_url,
        api_key="ollama",  # required but ignored by Ollama
        temperature=temperature,
        max_tokens=max_tokens,
        timeout=120,
        max_retries=2,
    )

Key parameters: keep temperature low (0.1–0.2) for deterministic agent behavior; set max_tokens high enough for multi-step reasoning traces; increase timeout because local inference on CPU or smaller GPUs can exceed the default 60 seconds.

Step 4: Define agents with explicit role prompts

CrewAI agents need clear, unambiguous role definitions. Local models follow instructions well but benefit from explicit formatting requirements. Define each agent in its own module for testability.

# agents.py
from crewai import Agent
from llm import get_local_llm

llm = get_local_llm()

researcher = Agent(
    role="Technical researcher",
    goal="Find accurate, up-to-date information on {topic} and cite sources",
    backstory=(
        "You are a meticulous researcher who verifies claims against primary sources. "
        "You never hallucinate URLs or statistics. If uncertain, you say so."
    ),
    llm=llm,
    verbose=True,
    allow_delegation=False,
    max_iter=3,
)

writer = Agent(
    role="Technical writer",
    goal="Produce a clear, well-structured summary from research notes",
    backstory=(
        "You write for software engineers. You use concrete examples, "
        "avoid fluff, and structure output with headings and code blocks where appropriate."
    ),
    llm=llm,
    verbose=True,
    allow_delegation=False,
    max_iter=2,
)

reviewer = Agent(
    role="Senior engineer reviewer",
    goal="Check the summary for technical accuracy, completeness, and clarity",
    backstory=(
        "You have 15 years of systems engineering experience. You catch subtle bugs, "
        "missing edge cases, and vague language. You output a pass/fail verdict and specific fixes."
    ),
    llm=llm,
    verbose=True,
    allow_delegation=False,
    max_iter=2,
)

Notes: max_iter caps the internal reasoning loop — critical for local models that can spiral. verbose=True prints the full reasoning trace to stdout, which is invaluable for debugging. allow_delegation=False keeps the flow linear and predictable for this tutorial.

Step 5: Define tasks with explicit output contracts

Tasks should specify exact output formats. Local models adhere to structure when you demand it in the expected_output field.

# tasks.py
from crewai import Task
from agents import researcher, writer, reviewer

research_task = Task(
    description=(
        "Research {topic} thoroughly. Focus on: current best practices, "
        "common pitfalls, and concrete code examples. Return findings as "
        "structured markdown with sections: Overview, Key Points, Code Examples, "
        "Gotchas, References. Each reference must include a verifiable URL."
    ),
    expected_output=(
        "Markdown document with exactly five sections: Overview, Key Points, "
        "Code Examples, Gotchas, References. No extra commentary."
    ),
    agent=researcher,
)

write_task = Task(
    description=(
        "Using the research output, write a concise technical guide for {topic}. "
        "Target audience: mid-level backend engineers. Include a working code snippet. "
        "Structure: Introduction, Prerequisites, Step-by-step, Verification, Troubleshooting."
    ),
    expected_output=(
        "Complete technical guide in markdown with the five sections above. "
        "Code blocks must be fenced with language tags. No placeholder text."
    ),
    agent=writer,
    context=[research_task],
)

review_task = Task(
    description=(
        "Review the technical guide for: technical accuracy, completeness, "
        "clarity, and runnable code. Output a JSON object with keys: "
        "verdict (pass/fail), issues (array of strings), fixes (array of strings). "
        "If verdict is fail, issues must be non-empty."
    ),
    expected_output=(
        "Valid JSON object with keys: verdict, issues, fixes. No markdown, no extra text."
    ),
    agent=reviewer,
    context=[write_task],
)

The context parameter chains task outputs — the writer sees research results, the reviewer sees the written guide. This is how CrewAI passes data between agents.

Step 6: Assemble and run the crew

Wire agents and tasks into a Crew with a sequential process. Sequential is the right default for local models — parallel execution multiplies VRAM pressure.

# main.py
from crewai import Crew, Process
from agents import researcher, writer, reviewer
from tasks import research_task, write_task, review_task

def run_crew(topic: str) -> str:
    crew = Crew(
        agents=[researcher, writer, reviewer],
        tasks=[research_task, write_task, review_task],
        process=Process.sequential,
        verbose=True,
        memory=False,  # disable for local models — adds latency and context pressure
    )

    result = crew.kickoff(inputs={"topic": topic})
    return result

if __name__ == "__main__":
    import sys
    topic = sys.argv[1] if len(sys.argv) > 1 else "async Python database connections with asyncpg"
    output = run_crew(topic)
    print("\n=== FINAL OUTPUT ===\n")
    print(output)

Run it:

python main.py "building a rate limiter in Go"

Expect 2–5 minutes per run on a 7B model with 8–12 GB VRAM. The verbose=True flag streams each agent’s reasoning, tool calls, and intermediate outputs — watch for loops or hallucinations.

Step 7: Verify success and iterate

A successful run produces three artifacts in sequence: research markdown, a technical guide, and a JSON review verdict. Check each:

1. Research output — five sections present, references have real URLs (spot-check one).
2. Guide output — compiles/runs if you copy the code snippet; no TODO or FIXME placeholders.
3. Review JSON — valid JSON, verdict is "pass" or "fail" with actionable issues and fixes.

If the reviewer returns fail, feed its fixes back into a second crew run. You can automate this with a simple loop:

# iterate.py
import json
from main import run_crew

def run_with_review(topic: str, max_rounds: int = 2) -> str:
    for round_num in range(max_rounds):
        print(f"\n=== Round {round_num + 1} ===")
        result = run_crew(topic)
        
        # Extract the review task output (last task)
        # CrewAI stores task outputs in result.tasks_output
        review_output = result.tasks_output[-1].raw
        
        try:
            review = json.loads(review_output)
        except json.JSONDecodeError:
            print("Review output not valid JSON, stopping")
            break
        
        if review.get("verdict") == "pass":
            print("Review passed")
            return result.tasks_output[-2].raw  # return the guide
        
        print(f"Review failed: {review['issues']}")
        print(f"Applying fixes: {review['fixes']}")
        
        # Append fixes to topic for next round
        topic += f"\n\nREVISION NOTES: {review['fixes']}"
    
    return "Max rounds reached without pass"

if __name__ == "__main__":
    import sys
    topic = sys.argv[1] if len(sys.argv) > 1 else "implementing a circuit breaker in Python"
    final = run_with_review(topic)
    print("\n=== FINAL GUIDE ===\n")
    print(final)

This pattern — generate, review, revise — is where local models shine. You pay zero marginal cost for iterations.

Step 8: Optimize for your hardware

Three levers matter most:

Context window: Qwen2.5 supports 32K context, but Ollama defaults to 4K. Increase it in your Modelfile if agents truncate:

# Modelfile
FROM qwen2.5:7b-instruct
PARAMETER num_ctx 16384
PARAMETER num_predict 4096

Build and use it:

ollama create qwen2.5:7b-instruct-16k -f Modelfile
# Update llm.py model name to "qwen2.5:7b-instruct-16k"

GPU offload: Ollama auto-detects Metal/CUDA/ROCm. Force full offload:

OLLAMA_NUM_GPU=999 ollama serve

Concurrency: For multiple simultaneous crews, run separate Ollama instances on different ports (OLLAMA_HOST=0.0.0.0:11435) and point each crew at its own endpoint. Local models don’t batch requests well — serialize or shard.

Step 9: Add tools for real-world utility

Agents that only reason are limited. Give them tools — web search, file I/O, code execution. CrewAI’s tools parameter accepts any LangChain tool. Example: a read-only filesystem tool for the researcher.

# tools.py
from langchain.tools import Tool
import os

def read_file(path: str) -> str:
    """Read a file from the local workspace. Path relative to project root."""
    base = os.path.abspath(".")
    target = os.path.abspath(os.path.join(base, path))
    if not target.startswith(base):
        return "Error: path traversal not allowed"
    try:
        with open(target, "r") as f:
            return f.read()[:8000]  # cap at 8K chars
    except Exception as e:
        return f"Error: {e}"

file_reader = Tool(
    name="read_file",
    func=read_file,
    description="Read a file from the project directory. Input: relative path.",
)

# Attach to researcher
researcher.tools = [file_reader]

Now the researcher can inspect existing codebases, configs, or docs. Add a write_file tool for the writer to emit artifacts directly.

Step 10: Production considerations

Running this in a CI/CD pipeline or scheduled job? A few hardening steps:

• Health check: Wrap ollama serve in a systemd unit or Docker container with a readiness probe (GET /api/tags).
• Model pinning: Tag the exact digest (qwen2.5:7b-instruct-q4_K_M@sha256:...) so updates don’t silently change behavior.
• Observability: Log each crew run’s inputs, outputs, latency, and token counts to a structured log (JSONL). Local inference has no built-in metering.
• Fallback: If you need guaranteed uptime, route to a hosted endpoint when Ollama is unhealthy. This is where a gateway like n4n.ai fits — one OpenAI-compatible endpoint that addresses 240+ models with automatic fallback when a provider is degraded, while preserving your routing directives and forwarding provider cache-control hints.

# fallback_llm.py
import os
from langchain_openai import ChatOpenAI

def get_llm_with_fallback() -> ChatOpenAI:
    # Try local first
    if os.getenv("USE_LOCAL", "true").lower() == "true":
        try:
            return get_local_llm()
        except Exception:
            pass  # fall through
    
    # Fallback to hosted (n4n.ai or any OpenAI-compatible)
    return ChatOpenAI(
        model=os.getenv("FALLBACK_MODEL", "qwen2.5-14b-instruct"),
        base_url=os.getenv("FALLBACK_BASE_URL", "https://api.n4n.ai/v1"),
        api_key=os.getenv("FALLBACK_API_KEY"),
        temperature=0.1,
        max_tokens=4096,
    )

This keeps your agent code unchanged — swap the LLM factory, not the crew.

You now have a complete, runnable local multi-agent pipeline: CrewAI agents driven by Qwen2.5 on Ollama, with structured tasks, review loops, tool access, and a fallback path. The same pattern scales to more agents, more complex tools, and larger models (Qwen2.5-32B on 24 GB VRAM, or Qwen2.5-72B with quantization on dual GPU). Start small, measure latency and quality, then expand.