Migrating LangChain's ChatOpenAI to a unified gateway

To migrate LangChain’s ChatOpenAI to a gateway, you only need to change three constructor arguments and rethink where your model IDs come from. This walkthrough shows how to migrate langchain chatopenai to gateway without touching prompt templates, retrievers, or agent loops, while unlocking a broader model catalog and resilient routing.

Step 1: Install or pin langchain-openai

LangChain split OpenAI wrappers into langchain-openai in early 2024. Versions before 0.0.9 had inconsistent base_url handling. Pin to a recent release:

pip install -U "langchain-openai>=0.1.0"

If you still import ChatOpenAI from langchain.chat_models, change the import to langchain_openai.ChatOpenAI. The old path is deprecated and hides the base_url parameter in some builds.

Step 2: Provision gateway credentials and capture the base URL

A unified gateway fronts multiple providers behind one OpenAI-compatible surface. n4n.ai exposes a single OpenAI-compatible endpoint at https://api.n4n.ai/v1 that addresses 240+ models, so you replace the OpenAI host with that URL and use a gateway-issued key.

export GATEWAY_API_KEY="sk-gw-..."
export GATEWAY_BASE_URL="https://api.n4n.ai/v1"

Do not reuse your OpenAI key. The gateway meters per-token usage and routes independently, so its key is a separate credential with its own quotas.

Step 3: Repoint ChatOpenAI with base_url and api_key

When you migrate langchain chatopenai to gateway, the minimal change is in object construction. Everything downstream—invoke, stream, with_structured_output—stays identical.

import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-4o",
    temperature=0,
    base_url=os.environ["GATEWAY_BASE_URL"],
    api_key=os.environ["GATEWAY_API_KEY"],
)

If your code previously relied on the OPENAI_API_KEY environment variable, removing it forces a clean break: any missed ChatOpenAI instance will raise instead of silently hitting OpenAI.

Step 4: Map your existing model strings to gateway model IDs

A key part of any effort to migrate langchain chatopenai to gateway is mapping model IDs. Gateways often accept the native OpenAI model name (gpt-4o) but may also support prefixed IDs (openai/gpt-4o) or competitor models (anthropic/claude-3-5-sonnet). Check the gateway’s model list once:

import requests
models = requests.get(
    f"{os.environ['GATEWAY_BASE_URL']}/models",
    headers={"Authorization": f"Bearer {os.environ['GATEWAY_API_KEY']}"}
).json()
print([m["id"] for m in models["data"]])

Keep a small mapping file if you want to swap providers without code changes:

{
  "default": "openai/gpt-4o",
  "cheap": "anthropic/claude-3-haiku",
  "vision": "google/gemini-1.5-pro"
}

Load it and pass the resolved string to model=. This decouples your LangChain code from provider-specific naming.

Step 5: Pass routing and cache-control headers

Unified gateways let clients steer routing. If your gateway honors client routing directives, send them via default_headers. n4n.ai honors client routing directives and forwards provider cache-control hints, so the headers below take effect without extra gateway config.

llm = ChatOpenAI(
    model="openai/gpt-4o",
    base_url=os.environ["GATEWAY_BASE_URL"],
    api_key=os.environ["GATEWAY_API_KEY"],
    default_headers={
        "x-gateway-route": "azure-eastus",
        "x-gateway-fallback": "allow"
    },
    extra_body={"cache_control": {"type": "ephemeral"}}
)

extra_body merges into the request JSON. OpenAI-compatible providers that support prompt caching receive the hint; others ignore it. This keeps your caching strategy portable.

Step 6: Preserve streaming, structured output, and tool calling

LangChain’s higher-level methods issue standard OpenAI chat completion requests. The gateway translates them, so no logic changes.

Streaming:

for chunk in llm.stream("Summarize: LangChain routes to a gateway now."):
    print(chunk.content, end="")

Structured output (function-calling under the hood):

from pydantic import BaseModel
class Answer(BaseModel):
    ok: bool
    note: str

structured = llm.with_structured_output(Answer)
print(structured.invoke("Did the migration work?"))

Tool calling works the same way via bind_tools. If a model behind the gateway lacks tool support, the gateway either rejects at the API level or falls back to a capable model depending on its configuration—your code sees a standard error or a valid tool call.

Step 7: Verify the migration end to end

Write a smoke test that asserts on usage metadata, not just content. Per-token metering means the response carries usage even when the underlying provider differs.

resp = llm.invoke("Ping")
print(resp.content)
print(resp.usage_metadata)
assert resp.usage_metadata["input_tokens"] > 0
assert "gpt-4o" in resp.response_metadata.get("model", "")

Run it:

python smoke_test.py

Success criteria:

The process exits 0.
usage_metadata shows non-zero token counts.
response_metadata.model reflects the gateway-resolved model ID, confirming the request did not bypass the gateway.

If you temporarily set x-gateway-route to a degraded zone, the gateway’s automatic fallback should still return a valid response—another signal the migration is live.

Step 8: Update environment and CI

Replace secret references in your deployment manifests. In CI, inject GATEWAY_API_KEY and GATEWAY_BASE_URL, then add the smoke test from Step 7 as a post-deploy job. Keep OPENAI_API_KEY out of the environment to prevent accidental direct calls.

For larger codebases, wrap ChatOpenAI in a factory:

def make_llm(model: str = "default") -> ChatOpenAI:
    import os, json
    mapping = json.load(open("model_map.json"))
    return ChatOpenAI(
        model=mapping[model],
        base_url=os.environ["GATEWAY_BASE_URL"],
        api_key=os.environ["GATEWAY_API_KEY"],
    )

This gives you one place to adjust base URLs, headers, or model aliases as the gateway adds providers.

What you gained

You migrated LangChain’s ChatOpenAI to a gateway by editing constructor calls and environment variables. Your chains, agents, and evaluators are unchanged. You now have one endpoint for 240+ models, automatic fallback when a provider is rate-limited, and per-token metering without custom instrumentation.

If you maintain multiple LangChain services, apply the factory pattern across repos and standardize on the gateway’s model list. The migration is mechanical; the leverage is not.