A CrewAI crew for automated SEO content audits

This crewai seo content audit example shows how to compose three specialized agents into a CrewAI crew that audits a blog post for SEO gaps. You will fetch raw HTML, evaluate meta tags and keyword density, and generate a prioritized remediation report—all without manual prompting spaghetti.

Step 1: Install dependencies and configure the LLM

CrewAI needs a chat model to drive agent reasoning. Install the framework plus HTTP and parsing libraries:

pip install crewai==0.30.0 requests beautifulsoup4

Export your API key for the model provider. For local testing, OpenAI works out of the box:

export OPENAI_API_KEY="sk-..."

If you later scale this to hundreds of pages, provider rate limits will bite. Point CrewAI’s LLM at an OpenAI-compatible endpoint such as n4n.ai to get automatic fallback when a provider is rate-limited or degraded, plus per-token metering. The base_url swap is the only change required:

from crewai import LLM

llm = LLM(
    model="openai/gpt-4o-mini",
    base_url="https://api.n4n.ai/v1",
    api_key="your-key",
)

Step 2: Build the content fetcher tool

The first agent should not reason about SEO; it should just return clean text. Wrap requests and BeautifulSoup in a CrewAI tool so the agent can call it deterministically.

from crewai.tools import tool
import requests
from bs4 import BeautifulSoup

@tool("Fetch and parse webpage")
def fetch_page(url: str) -> str:
    """Fetch a URL and return title, meta description, h1, and truncated body."""
    resp = requests.get(url, timeout=10, headers={"User-Agent": "Mozilla/5.0"})
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "html.parser")
    title = soup.title.string if soup.title else ""
    meta = soup.find("meta", attrs={"name": "description"})
    meta_desc = meta.get("content", "") if meta else ""
    h1 = soup.h1.get_text(strip=True) if soup.h1 else ""
    body = soup.get_text(separator=" ", strip=True)[:4000]
    return f"TITLE: {title}\nMETA_DESC: {meta_desc}\nH1: {h1}\nBODY: {body}"

Define the fetcher agent. Keep its goal narrow so the planner doesn’t overload it.

from crewai import Agent

fetch_agent = Agent(
    role="Content Fetcher",
    goal="Retrieve and normalize web page content for downstream analysis",
    backstory="Reliable scraping specialist who strips noise from HTML.",
    tools=[fetch_page],
    verbose=False,
    llm=llm,
)

Step 3: Define the SEO analysis agent

The auditor reads the fetcher’s output and applies concrete rules. We let the LLM judge semantic issues (thin content, keyword stuffing) but constrain it with an explicit checklist in the task description.

seo_agent = Agent(
    role="SEO Auditor",
    goal="Detect on-page SEO defects from fetched content",
    backstory="Senior technical SEO who thinks in crawl budgets and CTR.",
    verbose=False,
    llm=llm,
)

A separate tool is unnecessary here; the agent’s reasoning is the value. If you want reproducible scores, add a small Python function that computes keyword density and title length, then pass its output into the task context.

Step 4: Define the report writer agent

The final agent converts a bullet list of issues into an actionable markdown doc. Separating this from analysis prevents the auditor from softening findings to sound polite.

report_agent = Agent(
    role="Remediation Writer",
    goal="Produce a prioritized fix list from SEO audit findings",
    backstory="Editor who translates audit logs into ship-ready tickets.",
    verbose=False,
    llm=llm,
)

Step 5: Wire tasks and run the crew

Tasks declare context dependencies so CrewAI passes outputs forward. This crewai seo content audit example uses sequential processing—fetch, then audit, then report.

from crewai import Task, Crew, Process

fetch_task = Task(
    description="Fetch {url} and extract TITLE, META_DESC, H1, BODY.",
    expected_output="Structured text block with those four fields.",
    agent=fetch_agent,
)

audit_task = Task(
    description=(
        "Analyze the fetched content for: title length > 60 chars, "
        "missing/duplicate meta description, H1 absent, keyword '{keyword}' "
        "density < 0.5% or > 3%, body < 300 words. List each issue with severity."
    ),
    expected_output="Bullet list of issues tagged High/Medium/Low.",
    agent=seo_agent,
    context=[fetch_task],
)

report_task = Task(
    description="Turn the audit into a markdown report with sections by priority.",
    expected_output="Markdown with ## High, ## Medium, ## Low and exact fixes.",
    agent=report_agent,
    context=[audit_task],
)

crew = Crew(
    agents=[fetch_agent, seo_agent, report_agent],
    tasks=[fetch_task, audit_task, report_task],
    process=Process.sequential,
)

result = crew.kickoff(inputs={
    "url": "https://example.com/blog/old-post",
    "keyword": "llm gateway",
})

Running this crewai seo content audit example end-to-end takes under a minute for a single URL on a small model.

Step 6: Verify the output

Print result and confirm structure. For automated checks, assert the report contains priority headers and at least one issue:

text = result.raw if hasattr(result, "raw") else str(result)
assert "## High" in text or "## Medium" in text, "No issues captured"
assert "TITLE:" not in text, "Raw fetch leaked into final report"
print(text[:500])

If you see the fetcher’s raw TITLE: prefix in the final output, the context chaining broke—usually because a task expected_output mismatched. Fix by tightening the audit task description.

Step 7: Extend to a content pipeline

A single URL is a toy. Wrap the crew in a loop over your sitemap and write each report to disk:

import xml.etree.ElementTree as ET

urls = ET.parse("sitemap.xml").getroot()
for loc in urls.iter("{http://www.sitemaps.org/schemas/sitemap/0.9}loc"):
    out = crew.kickoff(inputs={"url": loc.text, "keyword": "inference"})
    with open(f"audit_{hash(loc.text)}.md", "w") as f:
        f.write(str(out))

When batching, set verbose=False and add retry logic around requests in the tool. The crewai seo content audit example above uses a 10-second timeout; bump it and cache responses to avoid re-fetching.

For cost control, the LLM call metering from a gateway becomes useful: you can attribute per-token spend to each audited URL. If you wired base_url to an OpenAI-compatible endpoint that honors client routing directives, you can pin cheap models for the fetcher (which needs no LLM) and reserve a stronger model for the auditor.

One more production note: CrewAI agents share state via task context, but they do not deduplicate across runs. Store audit hashes in a SQLite table to skip pages that haven’t changed since the last crawl. That turns the example into a real content-pipeline component rather than a one-shot script.