Coordinating AutoGen agent teams across GPT-5.1 and Gemini 3

Building an autogen agent team multi-model gpt-5.1 gemini 3 setup lets you pair OpenAI’s structured reasoning with Gemini’s wide-context synthesis in a single workflow. This guide shows how to wire that team with AutoGen’s agentchat API, using OpenAI-compatible model clients so you can swap providers without rewriting agent logic.

Step 1: Install the AutoGen v0.4 stack

AutoGen’s agentchat interface stabilized in the 0.4 line. Install the core conversation package and the model extension:

pip install autogen-agentchat==0.4.0 autogen-ext==0.4.0

Keep openai as a transitive dependency; the extension reuses its HTTP layer. If you pin versions in production, also pin openai<2.0 to avoid breaking the client.

Step 2: Configure model clients for GPT-5.1 and Gemini 3

AutoGen’s OpenAIChatCompletionClient speaks the OpenAI chat protocol. Any gateway that mirrors that protocol can serve non-OpenAI models. For a unified setup, point both clients at the same base URL and vary the model field.

import os
from autogen_ext.models.openai import OpenAIChatCompletionClient

API_KEY = os.environ["LLM_GATEWAY_KEY"]
BASE_URL = os.environ.get("LLM_GATEWAY_URL", "https://api.your-gateway/v1")

gpt51 = OpenAIChatCompletionClient(
    model="gpt-5.1",
    api_key=API_KEY,
    base_url=BASE_URL,
    temperature=0.2,
)

gemini3 = OpenAIChatCompletionClient(
    model="gemini-3",
    api_key=API_KEY,
    base_url=BASE_URL,
    temperature=0.4,
)

If you route through n4n.ai, a single OpenAI-compatible endpoint addresses both models and applies automatic fallback when a provider is rate-limited, so the same BASE_URL works for GPT-5.1 and Gemini 3 without code branches.

Test connectivity before building agents

A quick sanity call avoids debugging nested agent errors later:

resp = await gpt51.create([{"role": "user", "content": "ping"}])
print(resp.content, resp.usage)

Do the same for gemini3. If either raises ModelClientResponseError, fix auth or model string first.

Honor cache-control hints

Gemini and GPT differ in prompt caching. The client forwards cache_control markers if you pass them in request_options. Set extra_body on calls where long system prompts repeat:

gpt51 = OpenAIChatCompletionClient(
    model="gpt-5.1",
    api_key=API_KEY,
    base_url=BASE_URL,
    request_options={"extra_body": {"cache_control": {"type": "ephemeral"}}},
)

This matters when the planner’s system prompt is large and reused across turns.

Step 3: Define agent roles and system prompts

A two-model team needs clear separation of duties. Use GPT-5.1 for planning and critique; use Gemini 3 for retrieval-heavy drafting where its context window shines.

from autogen_agentchat.agents import AssistantAgent

planner = AssistantAgent(
    name="planner",
    model_client=gpt51,
    system_message=(
        "You decompose research tasks into ordered steps. "
        "Output only a JSON list of steps with 'action' and 'input' keys."
    ),
)

researcher = AssistantAgent(
    name="researcher",
    model_client=gemini3,
    system_message=(
        "You synthesize long documents into factual briefs. "
        "Cite sources inline as [src:n]. Never invent facts."
    ),
)

critic = AssistantAgent(
    name="critic",
    model_client=gpt51,
    system_message=(
        "You audit briefs for logical gaps and flag missing citations. "
        "If clean, reply with 'TERMINATE'."
    ),
)

Keep prompts tight. AutoGen passes the full conversation to each model, so a verbose system prompt multiplies token cost on every turn. Gemini 3 tolerates longer contexts, but GPT-5.1 will bill for every repeated token.

Step 4: Assemble the agent team

RoundRobinGroupChat cycles agents in order; SelectorGroupChat lets a model pick the next speaker. For a deterministic pipeline, round-robin is easier to debug.

from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_agentchat.conditions import TextMentionTermination

termination = TextMentionTermination("TERMINATE")

team = RoundRobinGroupChat(
    participants=[planner, researcher, critic],
    termination_condition=termination,
    max_turns=9,
)

If you need dynamic handoff, swap in SelectorGroupChat with a GPT-5.1 selector client. That adds latency but reduces wasted turns when the researcher finishes early.

from autogen_agentchat.teams import SelectorGroupChat

selector = SelectorGroupChat(
    participants=[planner, researcher, critic],
    model_client=gpt51,
    termination_condition=termination,
)

Step 5: Run a multi-model task

Kick off the team with a user message. AutoGen streams responses; we block on run_stream and collect the final message.

import asyncio

async def main():
    task = "Summarize the 2025 quantum error correction papers and propose 3 experiment ideas."
    stream = team.run_stream(task=task)
    async for event in stream:
        if isinstance(event, tuple):
            agent, msg = event
            print(f"{agent.name}: {msg.content[:200]}")
    print("Done")

asyncio.run(main())

Expected flow: planner emits JSON steps, researcher fills briefs from supplied documents, critic validates. When critic outputs “TERMINATE”, the loop ends. If max_turns hits first, you have a stuck agent—usually the researcher ignoring citation format.

Step 6: Verify success and inspect metering

Success means the critic returned a brief with zero open gaps and the team exited via termination, not max_turns. Print the last message and token counts:

async def verify():
    stream = team.run_stream(task="Summarize the 2025 QEC papers.")
    last = None
    async for ev in stream:
        last = ev
    if last and "TERMINATE" in str(last):
        print("Team terminated cleanly")
    # Actual usage lives on the client's last response
    print("GPT-5.1 last usage:", gpt51.last_usage)
    print("Gemini-3 last usage:", gemini3.last_usage)

Most OpenAI-compatible gateways return usage in the response body. If you use a gateway with per-token usage metering, parse the response’s usage field to attribute cost to each agent. That visibility is critical when Gemini 3 handles 100k-token inputs but GPT-5.1 does the final trim.

A concrete verification script can assert:

assert "TERMINATE" in critic_last_message
assert gemini3.last_usage.prompt_tokens > 1000  # long context exercised
print("Verification passed")

Operational caveats

Latency compounds in round-robin. Gemini 3 may take 3× longer on huge contexts than GPT-5.1 on short steps. Set max_turns aggressively and use TextMentionTermination to bail early.

Model drift is real. GPT-5.1 and Gemini 3 format JSON differently; enforce schemas with a post-parse step rather than trusting the agent. Wrap researcher output in a pydantic model before critic sees it.

If a provider degrades, the shared gateway’s fallback keeps the team running. Without it, catch ModelClientResponseError and rebuild the client with a backup base URL. AutoGen does not auto-switch models inside a running agent; you must recreate the AssistantAgent with the new client.

The autogen agent team multi-model gpt-5.1 gemini 3 pattern scales to more models by adding clients and participants, but keep the speaker graph acyclic to avoid infinite loops. Treat the team as a pipeline, not a debate club, unless you explicitly want adversarial dynamics.