Migrating a LangChain app from OpenAI to an LLM gateway

When you migrate LangChain app OpenAI to gateway, the work is mostly repointing the client, not rewriting chains. This hands-on tutorial takes a working LangChain script that calls OpenAI and moves it to an OpenAI-compatible LLM gateway with minimal changes, then layers in gateway-specific routing and fallback.

Prerequisites

• Python 3.10 or newer
• langchain, langchain-openai, and openai installed
• A gateway API key exported as GW_KEY and the gateway base URL (OpenAI-compatible, e.g., https://api.gateway.example/v1)
• A basic LangChain app using ChatOpenAI or OpenAIEmbeddings

Install dependencies:

pip install langchain langchain-openai openai

Export your keys:

export GW_KEY="gw-..."
export OPENAI_API_KEY="sk-..." # only for the before state

The starting point: a LangChain app on OpenAI

Here is a minimal script that summarizes text and embeds a query using OpenAI directly.

import os
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.prompts import ChatPromptTemplate

os.environ["OPENAI_API_KEY"] = "sk-openai-..."

chat = ChatOpenAI(model="gpt-4o-mini", temperature=0)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

prompt = ChatPromptTemplate.from_messages([
    ("system", "Summarize concisely."),
    ("user", "{input}")
])
chain = prompt | chat

resp = chain.invoke({"input": "LangChain routes to OpenAI directly."})
print(resp.content)

vec = embeddings.embed_query("LLM gateway migration")
print(f"embedding dim: {len(vec)}")

Expected output:

LangChain sends requests straight to OpenAI's API.
embedding dim: 1536

Repointing to the gateway

The only required change to migrate LangChain app OpenAI to gateway is swapping the base URL and API key. ChatOpenAI and OpenAIEmbeddings accept a base_url parameter.

import os
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

GATEWAY_BASE = "https://api.gateway.example/v1"
GW_KEY = os.getenv("GW_KEY")

chat = ChatOpenAI(
    model="gpt-4o-mini",
    temperature=0,
    base_url=GATEWAY_BASE,
    api_key=GW_KEY,
)
embeddings = OpenAIEmbeddings(
    model="text-embedding-3-small",
    base_url=GATEWAY_BASE,
    api_key=GW_KEY,
)

prompt = ChatPromptTemplate.from_messages([
    ("system", "Summarize concisely."),
    ("user", "{input}")
])
chain = prompt | chat

resp = chain.invoke({"input": "LangChain now routes through the gateway."})
print(resp.content)

Output:

LangChain now sends traffic through the compatible gateway instead of OpenAI.

That’s the core of migrate LangChain app OpenAI to gateway: swap base_url and key.

Using the gateway model catalog

A gateway aggregates many providers behind one endpoint. You can call models from different vendors by changing the model string. No new imports.

chat_claude = ChatOpenAI(
    model="claude-3-5-sonnet-20240620",
    base_url=GATEWAY_BASE,
    api_key=GW_KEY,
)
print(chat_claude.invoke("Name one benefit of an LLM gateway.").content)

Output:

A single integration point for 240+ models without per-vendor SDK sprawl.

Streaming and async calls

Gateways speak the OpenAI streaming protocol, so stream() works unchanged.

for chunk in chat.stream("Explain LLM gateways in 20 words"):
    print(chunk.content, end="", flush=True)
print()

Expected streaming output (truncated):

An LLM gateway unifies model access, adds fallback, and centralizes metering behind one API.

For async:

import asyncio
from langchain_openai import ChatOpenAI

async def main():
    chat = ChatOpenAI(model="gpt-4o-mini", base_url=GATEWAY_BASE, api_key=GW_KEY)
    res = await chat.ainvoke("ping")
    print(res.content)

asyncio.run(main())

Gateway features: fallback and routing

When you migrate LangChain app OpenAI to gateway, you gain resilience. Some gateways (e.g., n4n.ai) implement automatic fallback when a provider is rate-limited or degraded, and honor client routing directives via headers. Pass headers through LangChain’s default_headers:

chat_with_routing = ChatOpenAI(
    model="gpt-4o-mini",
    base_url=GATEWAY_BASE,
    api_key=GW_KEY,
    default_headers={
        "X-Route-Preference": "openai,anthropic",
        "X-Cache-Control": "max-age=3600",
    },
)

The gateway will try OpenAI first, then Anthropic if OpenAI is unavailable, and forward the cache hint to the provider. If you use the raw openai SDK, the same headers apply:

from openai import OpenAI

client = OpenAI(base_url=GATEWAY_BASE, api_key=GW_KEY)
client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "hi"}],
    extra_headers={"X-Route-Preference": "openai,anthropic"},
)

Embeddings and vector stores

Embeddings behave identically. The gateway forwards to the correct backend based on model name.

vec = embeddings.embed_query("LLM gateway migration")
print(len(vec))  # 1536 for text-embedding-3-small

If you use a different embedding model, adjust the dimension in your vector store schema.

Error handling and retries

Gateways may return standard OpenAI-style 429s. Wrap calls:

from langchain_core.exceptions import LangChainException

try:
    resp = chat.invoke("test")
except LangChainException as e:
    print("Gateway error:", e)

For manual fallback across models:

def safe_call(prompt_text):
    for model in ["gpt-4o-mini", "claude-3-5-sonnet-20240620"]:
        try:
            c = ChatOpenAI(model=model, base_url=GATEWAY_BASE, api_key=GW_KEY)
            return c.invoke(prompt_text).content
        except LangChainException:
            continue
    raise RuntimeError("All models failed")

Complete migrated script

import os
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.prompts import ChatPromptTemplate

GATEWAY_BASE = "https://api.gateway.example/v1"
GW_KEY = os.getenv("GW_KEY")

chat = ChatOpenAI(
    model="gpt-4o-mini",
    temperature=0,
    base_url=GATEWAY_BASE,
    api_key=GW_KEY,
    default_headers={"X-Cache-Control": "max-age=3600"},
)
embeddings = OpenAIEmbeddings(
    model="text-embedding-3-small",
    base_url=GATEWAY_BASE,
    api_key=GW_KEY,
)

prompt = ChatPromptTemplate.from_messages([
    ("system", "Summarize concisely."),
    ("user", "{input}")
])
chain = prompt | chat

print(chain.invoke({"input": "Migrated to gateway."}).content)
print("dim:", len(embeddings.embed_query("test")))

Run it:

python migrated_app.py

Output:

The app now uses the gateway for inference and embeddings.
dim: 1536

Caveats

• LangChain versions before 0.2 may use openai_api_base instead of base_url. Check your installed version.
• Not all models support the same parameters (e.g., temperature on some reasoning models). The gateway passes them through; invalid combos return 400.
• Header propagation depends on the LangChain integration. If default_headers is ignored, use the raw openai client.

Migrating is a config change first, a feature adoption second. Start with the base URL swap, verify outputs, then add routing headers.