Deploying agents behind an OpenAI-compatible gateway

Setting up an OpenAI-compatible API gateway for agents lets you swap model providers, absorb rate limits, and centralize credential handling without touching agent business logic. This guide lays out an ordered deployment path we’ve used for production agent services: from endpoint configuration to routing rules, streaming, cache propagation, and usage metering. You should already have an agent that speaks the OpenAI chat completions protocol.

1. Inventory the agent’s model requirements

Before you route anything, write down exactly what the agent calls. Most agents need one or more of these:

• Function/tool calling
• JSON mode or structured output
• Vision inputs
• Context windows larger than 32K tokens
• Low-latency responses for interactive loops

A gateway can’t paper over missing capabilities. If your agent sends tools and the upstream model ignores them, you’ll get silent failures. Check the model matrix and encode the minimum bar in config.

REQUIRED_CAPS = {
    "tools": True,
    "json_mode": True,
    "vision": False,
    "min_context": 128_000,
}

Tradeoff: restricting to models that meet all criteria shrinks your fallback pool. Decide deliberately which capabilities are deal-breakers versus nice-to-haves.

2. Repoint the agent at a single gateway endpoint

The whole point of an OpenAI-compatible API gateway for agents is that the client code doesn’t change. Set base_url from an environment variable and keep the request shape identical to OpenAI’s.

from openai import OpenAI
import os

client = OpenAI(
    base_url=os.environ["GATEWAY_BASE_URL"],  # e.g. https://gw.internal/v1
    api_key=os.environ["GATEWAY_KEY"],
)

resp = client.chat.completions.create(
    model="router-gpt4o",  # logical name resolved by gateway
    messages=[{"role": "user", "content": "Plan a deploy"}],
)

If you’re using the TypeScript SDK, the same baseURL swap applies. Do not branch on provider inside the agent. Let the gateway own that decision.

Common pitfall: hardcoding api.openai.com in a dozen lambda functions. Search your repo for openai.com and replace with the env var before proceeding.

3. Define routing and fallback rules

A gateway maps logical model names to upstream providers. Keep the agent’s requested model name stable; shift the backend underneath it.

{
  "routes": [
    {"match": {"model": "router-gpt4o"}, "upstream": "openai"},
    {"match": {"model": "router-sonnet"}, "upstream": "anthropic"},
    {"match": {"model": "router-fast"}, "upstream": "together"}
  ],
  "fallback": {
    "on_status": [429, 500, 503],
    "try": ["openai", "azure-openai", "together"]
  }
}

If you use a managed service such as n4n.ai, automatic fallback when a provider is rate-limited or degraded is built in; self-hosted setups need explicit retry-with-backoff and a circuit breaker. Either way, define the fallback order by cost and latency, not just availability.

Tradeoff: fallback can change model behavior mid-session. A tool-calling agent that quietly switches from GPT-4o to Mixtral may lose argument fidelity. Log the resolved upstream on every response so you can trace anomalies.

4. Verify streaming and tool passthrough

Agents rarely block on a full response. They stream tokens and interleave tool calls. Your gateway must pass SSE bytes unchanged, not buffer them.

const res = await fetch(`${BASE}/v1/chat/completions`, {
  method: "POST",
  headers: { "content-type": "application/json", authorization: `Bearer ${key}` },
  body: JSON.stringify({ model: "router-sonnet", messages, stream: true, tools }),
});
const reader = res.body!.getReader();
const decoder = new TextDecoder();
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  // parse `data:` lines, forward to agent loop
  processChunk(decoder.decode(value));
}

Test with curl using --no-buffer to confirm chunks arrive incrementally. If you see a single payload after a long delay, the gateway is buffering—fix that before launch.

Pitfall: some gateways rewrite finish_reason on tool calls. Ensure finish_reason: "tool_calls" survives the hop, or your agent loop will misclassify the turn.

5. Forward cache-control hints

Prompt caching cuts cost and latency on long system prompts and repeated tool schemas. Providers expose this via headers or special body fields. An OpenAI-compatible API gateway for agents should honor client routing directives and forward provider cache-control hints rather than stripping them.

client.chat.completions.create(
    model="router-sonnet",
    messages=SYSTEM_AND_TOOLS,
    extra_headers={"anthropic-beta": "prompt-caching-2024-07-31"},
)

If your gateway doesn’t support arbitrary header forwarding, you’ll either leak provider-specific quirks into the agent or lose the cache entirely. Validate by sending the same request twice and checking upstream token counts drop on the second call.

Tradeoff: cached prompts can stick around longer than you expect. Set explicit TTLs where the provider allows, and never cache ephemeral user data.

6. Meter usage per token

Agents burn tokens in tight loops; a runaway replanner can spend $50 in minutes. The gateway should return standard usage objects and aggregate them.

resp = client.chat.completions.create(model="router-fast", messages=msgs)
print(resp.usage.prompt_tokens, resp.usage.completion_tokens)

Centralize metering at the gateway so every agent inherits it. Per-token usage metering across heterogeneous models requires normalizing by a cost table you maintain. Don’t trust the agent to self-report; pull from the gateway’s response or logs.

Pitfall: ignoring cached token counts. prompt_tokens may include prompt_tokens_details.cached_tokens. If your budget alert sums only completion tokens, you’ll misattribute savings.

7. Choose a deployment topology

Put the gateway in the same network boundary as the agent. A sidecar container in the same pod beats a public endpoint for latency and secrecy.

# docker-compose snippet
services:
  agent:
    image: my-agent:1.4
    environment:
      GATEWAY_BASE_URL: http://gateway:8080/v1
  gateway:
    image: litellm:latest
    ports:
      - "8080:8080"

If you run multiple agents, a shared gateway service simplifies key rotation and routing updates. The OpenAI-compatible API gateway for agents pattern scales horizontally; statelessness is the goal.

Tradeoff: a shared gateway is a single point of failure. Run at least two replicas behind a load balancer and health-check the /v1/models endpoint.

8. Harden the edges

• Timeouts: set client timeout to 30–60s for agent steps; providers vary wildly.
• Retries: only retry idempotent failures (429, 503). Never retry on 400.
• Secrets: gateway holds provider keys; agents hold only the gateway key. Rotate the latter via env reload.
• Model drift: when you remap router-gpt4o to a new version, bump the logical name (router-gpt4o-2025-01) to avoid silent behavior shifts.

9. Validate end to end

Run a scripted agent task against the gateway with fallback forced by blocking the primary upstream. Confirm:

1. Request routed to secondary.
2. Streaming chunks arrived in order.
3. Tool call parsed correctly.
4. Usage logged with cached token detail.
5. Cache header forwarded (second call cheaper).

If any step fails, fix at the gateway layer, not in the agent. The agent should remain blissfully unaware of provider mechanics.

Common pitfalls summary

• Agent code contains provider conditionals—defeats the gateway.
• Gateway buffers streams, breaking interactive agents.
• Fallback changes model without logging, causing mysterious regressions.
• Cache headers stripped, inflating cost on long system prompts.
• Metering ignores cached tokens, masking true spend.

An OpenAI-compatible API gateway for agents is not a luxury; it’s the seam that keeps provider churn from becoming code churn. Build the routing and metering once, and let the agents focus on the work.