Building a reliable document pipeline requires separation of concerns. This guide shows how to assemble an autogen agent team report generation qa system that researches a topic, drafts a report, and independently audits it for accuracy and style.
Step 1: Install dependencies and configure the LLM endpoint
Install the classic AutoGen library (v0.2.x) which provides GroupChat primitives. The newer autogen-agentchat package changes the API; the patterns below use the stable pyautogen interface that most production code still targets.
pip install pyautogen==0.2.32Point AutoGen at an OpenAI-compatible endpoint. If you want a single endpoint that fronts 240+ models with automatic fallback when a provider is degraded, set the base_url to that gateway. For example, n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models and handles provider failover transparently.
import autogen
llm_config = {
"config_list": [
{
"model": "gpt-4o-mini",
"api_key": "YOUR_KEY",
"base_url": "https://api.n4n.ai/v1", # optional gateway
"api_type": "openai",
}
],
"temperature": 0.2,
"cache_seed": 42, # enables local caching of responses
}If you run locally with OpenAI directly, drop the base_url. The cache_seed makes runs deterministic and cuts cost during development. AutoGen caches by hashing the prompt + seed; re-running the same brief returns the cached completion without a network call.
Configuring multiple models per agent
You can give each agent a different model by passing a separate llm_config with its own config_list. A common setup: Researcher on a fast cheap model, Writer on a flagship model, QA on a medium model. The gateway forwards provider cache-control hints, so long system prompts aren’t re-billed every round.
Step 2: Define agent roles for the autogen agent team report generation qa flow
We need three specialized assistants: a Researcher, a Writer, and a QA Reviewer. Each gets a tight system message. Vague roles produce vague documents.
researcher = autogen.AssistantAgent(
name="Researcher",
llm_config=llm_config,
system_message=(
"You gather facts from the provided dataset or context. "
"Output only bullet points of verified facts with source references. "
"Do not write prose."
),
)
writer = autogen.AssistantAgent(
name="Writer",
llm_config=llm_config,
system_message=(
"You convert the Researcher's bullet points into a clean Markdown report. "
"Use headings, avoid speculation, cite sources inline."
),
)
qa = autogen.AssistantAgent(
name="QA",
llm_config=llm_config,
system_message=(
"You review the Markdown report against the Researcher's facts. "
"Return JSON: {\"pass\": bool, \"issues\": [str], \"score\": int}. "
"Be strict about unsupported claims."
),
)The user proxy drives the conversation and can execute code if needed, but for a document pipeline we keep it silent.
user = autogen.UserProxyAgent(
name="User",
human_input_mode="NEVER",
code_execution_config=False,
)Prompt design tips
- Researcher must not editorialize. Bullet points keep the Writer honest.
- Writer system message should forbid inventing sources. Instruct: “If a fact lacks a source, omit it.”
- QA must output parseable JSON. Specify the schema explicitly; LLMs follow structured constraints better when the format is in the system prompt.
Step 3: Construct the group chat and manager
AutoGen’s GroupChat cycles through speakers. We enforce order: Researcher → Writer → QA → Researcher (if QA fails). Use allowed_or_disallowed_speaker_transitions for strict flow, or simpler: set speaker_selection_method="round_robin" and max_round=6.
group = autogen.GroupChat(
agents=[user, researcher, writer, qa],
messages=[],
max_round=6,
speaker_selection_method="round_robin",
allow_repeat_speaker=False,
)
manager = autogen.GroupChatManager(group, llm_config=llm_config)Round-robin ensures each agent speaks in registration order. With allow_repeat_speaker=False, the same agent won’t talk twice consecutively. The user agent is first but stays silent because human_input_mode="NEVER" and no code execution.
Why not auto speaker selection?
LLM-based speaker selection adds latency and randomness. For a fixed pipeline, round-robin is deterministic and debuggable. Save the smart selection for open-ended brainstorming.
Step 4: Run the report generation task
Initiate the chat with a concrete brief and data. The user proxy sends the first message.
brief = """
Topic: "Edge caching strategies for LLM gateways"
Context data:
- Gateway X supports provider cache-control hints (source: internal doc).
- Fallback reduces p99 latency spikes during provider outages (source: ops log).
- Per-token metering enables cost attribution (source: billing API).
Produce a 300-word report.
"""
user.initiate_chat(manager, message=brief)The Researcher extracts facts, Writer drafts, QA scores. After max_round, the manager terminates.
To capture the final report, hook the chat:
def extract_final_report(messages):
# last message from Writer before QA
for msg in reversed(messages):
if msg["name"] == "Writer":
return msg["content"]
return None
history = group.messages
report = extract_final_report(history)
print(report)Feeding real data
In practice, load context from a file or database. Inject it into the brief string. Keep context under the model’s context window; if larger, chunk it and let the Researcher summarize per chunk.
Step 5: Implement strict QA gating
The QA agent returns JSON. Parse it and decide whether to accept or re-run.
import json
def get_qa_result(messages):
for msg in reversed(messages):
if msg["name"] == "QA":
try:
return json.loads(msg["content"])
except json.JSONDecodeError:
return {"pass": False, "issues": ["QA not valid JSON"], "score": 0}
return {"pass": False, "issues": ["No QA"], "score": 0}
qa_result = get_qa_result(group.messages)
if not qa_result["pass"]:
print("QA failed:", qa_result["issues"])
# trigger another group chat with QA feedback appended
user.initiate_chat(manager, message=f"QA issues: {qa_result['issues']}. Revise.")This loop creates an autogen agent team report generation qa feedback cycle. In production, cap revisions to avoid infinite loops.
Structured output parsing
Wrap json.loads in a retry that asks the QA agent to re-emit if malformed. AutoGen doesn’t natively validate JSON, so you own that logic.
Step 6: Verify success
Success means: a Markdown report exists, QA passed, and citations match sources. Write a small test:
def verify_pipeline(messages):
report = extract_final_report(messages)
qa_res = get_qa_result(messages)
assert report is not None, "No report produced"
assert qa_res["pass"] is True, f"QA failed: {qa_res['issues']}"
assert "source:" in report.lower(), "Report missing citations"
return True
assert verify_pipeline(group.messages)
print("Pipeline OK")Run the script. If assertions pass, you have a working autogen agent team report generation qa system.
Manual inspection
Always read the first report. LLMs can pass QA while still being bland. Use the score field to track quality trends across runs.
Step 7: Production hardening
For real workloads, externalize the LLM config and add retry logic. If you use a gateway that honors client routing directives, you can pin the Researcher to a cheap model and the Writer to a stronger one via model overrides per agent. The same OpenAI-compatible endpoint can forward provider cache-control hints, reducing repeated token cost on long contexts.
Keep max_round low (4–8). Longer conversations drift. Log group.messages to JSON for audit.
If you need to swap models mid-flight, AutoGen’s config_list supports multiple entries; the gateway’s automatic fallback covers rate limits without code changes.
That’s the whole pattern. Build the team, enforce order, gate on QA, verify.