AutoGen agents OpenAI-compatible API gateway deployments let you keep agent logic unchanged while swapping the underlying model provider at runtime. This guide shows how to stand up a multi-agent AutoGen system that talks to a single /v1/chat/completions endpoint, so you get provider fallback, per-token metering, and centralized cache control without touching agent code.
Step 1: Install and pin AutoGen
Use a clean virtual environment. The examples below target pyautogen 0.2.32, which is stable and widely deployed.
python -m venv .venv
source .venv/bin/activate
pip install pyautogen==0.2.32Set two environment variables: the gateway base URL and a gateway API key. AutoGen reads these from the config list, not from OPENAI_API_KEY, so name them explicitly.
export GATEWAY_BASE_URL="https://gateway.example.com/v1"
export GATEWAY_API_KEY="sk-gw-xxxxxxxx"If you run a local gateway for testing, point GATEWAY_BASE_URL at http://localhost:8080/v1.
Step 2: Define the client configuration
AutoGen’s OpenAIWrapper accepts a config_list where each entry is a full OpenAI-compatible client spec. The base_url field is the key to routing through the gateway.
import os
from autogen import config_list_from_json, OpenAIWrapper
config_list = [
{
"model": "gpt-4o",
"base_url": os.environ["GATEWAY_BASE_URL"],
"api_key": os.environ["GATEWAY_API_KEY"],
"api_type": "openai",
"extra_headers": {
"X-Route-Preference": "openai,anthropic"
}
}
]
wrapper = OpenAIWrapper(config_list=config_list)The extra_headers field is passed through to the gateway. A gateway that honors client routing directives will use X-Route-Preference to bias provider selection. Note that AutoGen does not validate these headers; they are opaque to the agent framework.
Step 3: Build a minimal two-agent system
We will use a UserProxyAgent to simulate a task and an AssistantAgent to solve it. The llm_config references the config_list from Step 2.
from autogen import UserProxyAgent, AssistantAgent, initiate_chat
assistant = AssistantAgent(
name="assistant",
llm_config={"config_list": config_list, "cache_seed": 42},
)
user_proxy = UserProxyAgent(
name="user_proxy",
human_input_mode="NEVER",
code_execution_config=False,
is_termination_msg=lambda m: "TERMINATE" in m.get("content", ""),
)
chat_result = user_proxy.initiate_chat(
assistant,
message="Write a Python function that flattens a nested list. TERMINATE after the code block.",
)
print(chat_result.summary)Run the script. If the gateway is reachable, you will see the assistant emit a function and the conversation end when it prints TERMINATE.
Step 4: Route different agents to different models
The AutoGen agents OpenAI-compatible API gateway pattern shines when each agent calls a different model behind the same endpoint. A gateway like n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models, so you just change the model string per agent.
config_assistant = [
{
"model": "gpt-4o",
"base_url": os.environ["GATEWAY_BASE_URL"],
"api_key": os.environ["GATEWAY_API_KEY"],
}
]
config_critic = [
{
"model": "claude-3-5-sonnet",
"base_url": os.environ["GATEWAY_BASE_URL"],
"api_key": os.environ["GATEWAY_API_KEY"],
}
]
assistant = AssistantAgent(
name="coder",
llm_config={"config_list": config_assistant},
)
critic = AssistantAgent(
name="critic",
llm_config={"config_list": config_critic},
system_message="You are a senior reviewer. Critique the code for edge cases.",
)Now initiate_chat between user_proxy and assistant, then forward the result to critic. Because both configs point at the same base_url, the gateway routes to the correct backend by model name. No agent code changes when you later swap claude-3-5-sonnet for mistral-large.
Step 5: Enable fallback and cache control
Provider outages are routine. A gateway that provides automatic fallback when a provider is rate-limited or degraded removes the need for retry loops in agent code. You still should set sensible timeouts in AutoGen.
config_with_timeout = [
{
"model": "gpt-4o",
"base_url": os.environ["GATEWAY_BASE_URL"],
"api_key": os.environ["GATEWAY_API_KEY"],
"timeout": 30,
"max_retries": 3,
}
]For cache control, some gateways such as n4n.ai honor client routing directives and forward provider cache-control hints. In AutoGen, pass them via extra_body or extra_headers depending on gateway spec. Example using extra_headers:
config_cached = [
{
"model": "gpt-4o",
"base_url": os.environ["GATEWAY_BASE_URL"],
"api_key": os.environ["GATEWAY_API_KEY"],
"extra_headers": {
"X-Cache-Control": "ephemeral"
}
}
]If the gateway forwards the hint to a provider that supports prompt caching, you reduce repeated token costs on long system prompts. Verify the gateway returns a usage object with cache_creation_tokens or similar.
Step 6: Run and verify success
Create run_agents.py with the full assembly from Steps 2–5. Execute:
python run_agents.pySuccess criteria:
- The script exits without
APIConnectionError. - The assistant produces a code block.
- The critic agent returns a non-empty review.
- The gateway access log shows one request per agent turn with the expected
modelvalues.
To inspect token metering, print the raw response from the wrapper:
response = wrapper.create(
messages=[{"role": "user", "content": "ping"}],
model="gpt-4o",
)
print(response.usage)If the gateway emits per-token usage metering, you will see prompt_tokens, completion_tokens, and total_tokens. Pipe these to your observability stack.
Step 7: Production hardening
AutoGen blocks on initiate_chat by default. For concurrent agent sessions, wrap each in a thread or use the async API (autogen.agentchat.contrib.async_chat). Set max_consecutive_auto_reply to prevent runaway loops:
assistant = AssistantAgent(
name="coder",
llm_config={"config_list": config_assistant},
max_consecutive_auto_reply=5,
)For long-running workflows, externalize config_list to a JSON file and load it with config_list_from_json("config.json"). This keeps secrets out of source control and lets ops rotate the gateway key without a deploy.
[
{
"model": "gpt-4o",
"base_url": "https://gateway.example.com/v1",
"api_key": "sk-gw-xxxxxxxx",
"timeout": 30
}
]from autogen import config_list_from_json
config_list = config_list_from_json("config.json")When you run the AutoGen agents OpenAI-compatible API gateway setup in production, monitor the gateway’s latency percentiles. AutoGen’s default 60-second timeout is too high for interactive agents; drop it to 15–20 seconds and let the gateway’s fallback handle provider hiccups.
Step 8: Debugging common failures
401 from gateway: Check that api_key is sent as Authorization: Bearer. AutoGen does this automatically when api_key is in the config entry.
Model not found: The gateway may not have the model alias you used. List available models via GET /v1/models and match the exact string.
Agents loop forever: Set is_termination_msg strictly. Without it, the critic’s reply triggers another assistant turn.
Cache hint ignored: Not all providers accept cache headers. The gateway may strip unknown headers. Test with a direct curl:
curl -X POST "$GATEWAY_BASE_URL/chat/completions" \
-H "Authorization: Bearer $GATEWAY_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"gpt-4o","messages":[{"role":"user","content":"hi"}],"extra_headers":{"X-Cache-Control":"ephemeral"}}'If the response lacks cache fields, the gateway or provider does not support it.
Wrap-up
The AutoGen agents OpenAI-compatible API gateway approach decouples agent logic from model infrastructure. You write agents once, point them at a single endpoint, and gain routing, fallback, and metering. The code above is runnable today against any compliant gateway; adapt the header names to your provider’s spec and ship.