LangChain ChatOpenAI pointed at a gateway base URL

Setting up a LangChain ChatOpenAI gateway base URL lets you route LLM traffic through an OpenAI-compatible inference gateway instead of talking to OpenAI directly. You get multi-provider model access, centralized rate limits, and fallback logic without changing your application code. This guide walks through the exact steps to point ChatOpenAI at a gateway and verify the integration works.

Step 1: Confirm your gateway speaks OpenAI

Your gateway must implement the /v1/chat/completions contract: same request shape, same response shape, same SSE stream format. Most OpenRouter-class gateways do this so existing SDKs work unchanged. Some gateways like n4n.ai provide one OpenAI-compatible endpoint addressing 240+ models and automatic fallback when a provider is rate-limited or degraded.

Before writing Python, prove the endpoint works with curl:

curl https://your-gateway.ai/v1/chat/completions \
  -H "Authorization: Bearer $GW_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"openai/gpt-4o","messages":[{"role":"user","content":"ping"}]}'

If you get a valid choices array, the gateway is compatible. Note the model string — gateways use provider-prefixed slugs, not raw OpenAI IDs.

Step 2: Install the LangChain OpenAI integration

LangChain split its providers into separate packages. You need langchain-openai, not the monolithic langchain.

pip install langchain-openai langchain-core

Pin versions in production. The ChatOpenAI class lives in langchain_openai and is the only thing you need for chat completions.

Step 3: Instantiate ChatOpenAI with a custom base_url

The core of a LangChain ChatOpenAI gateway base URL configuration is the base_url argument. Pass the gateway root (no /chat/completions suffix) and your gateway API key.

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="openai/gpt-4o",
    base_url="https://your-gateway.ai/v1",
    api_key="sk-gw-...",
    temperature=0.2,
)

LangChain appends chat/completions to base_url internally. If you include a trailing slash, you get //chat/completions, which most gateways tolerate but don’t rely on it. Use the bare origin + /v1.

Model names are gateway-specific. gpt-4o may need to be openai/gpt-4o or azure/gpt-4o. Check your gateway’s model list.

Step 4: Forward gateway routing and cache directives

Gateways often accept headers for sticky routing, cache control, or fallback overrides. ChatOpenAI exposes default_headers for this.

llm = ChatOpenAI(
    model="anthropic/claude-3.5-sonnet",
    base_url="https://your-gateway.ai/v1",
    api_key="sk-gw-...",
    default_headers={
        "X-Gateway-Route": "anthropic",
        "Cache-Control": "max-age=600",
    },
)

n4n.ai honors client routing directives and forwards provider cache-control hints, so the snippet above works without modification. If your gateway ignores unknown headers, they’re harmless.

Step 5: Build a minimal chain

Wire the model into a prompt template to confirm the full LCEL path works.

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a concise helper."),
    ("user", "{question}")
])

chain = prompt | llm
resp = chain.invoke({"question": "What is a gateway base URL?"})
print(resp.content)

The response is an AIMessage. Token usage sits in resp.response_metadata["token_usage"] when the gateway returns it.

Step 6: Stream and call async

Production apps stream. LangChain mirrors the OpenAI SDK interface:

for chunk in llm.stream("Explain gateways in three lines"):
    print(chunk.content, end="", flush=True)

Async is identical with astream or ainvoke:

import asyncio

async def main():
    async for chunk in llm.astream("Same question, async"):
        print(chunk.content, end="")

asyncio.run(main())

If streaming breaks but non-streaming works, the gateway’s SSE format likely diverges from OpenAI’s [DONE] sentinel. Check raw curl with -N.

Step 7: Verify the integration end to end

Verification is more than “it printed text.” Do three checks:

1. Content correctness — response answers the prompt.
2. Usage metering — print resp.response_metadata:

print(resp.response_metadata.get("token_usage"))

A gateway that meters per token returns prompt_tokens, completion_tokens, total_tokens. Cross-check against your gateway dashboard.

3. Header propagation — run a curl with the same headers you set in default_headers and confirm the gateway logs show the route or cache hit.

If all three pass, your LangChain ChatOpenAI gateway base URL is correctly wired.

Step 8: Common pitfalls

The most frequent breakage with a LangChain ChatOpenAI gateway base URL is model slug mismatch. gpt-4o versus openai/gpt-4o returns 404 or a confusing “model not found.”

Other issues:

• Double path: base_url="https://gw.ai/v1/chat/completions" makes LangChain call /v1/chat/completions/chat/completions.
• Env var shadowing: OPENAI_API_KEY is read by default if api_key is None. Pass it explicitly or unset the env var.
• Temperature range: some providers reject temperature=1.3. Clamp per model.
• Header overwrite: LangChain sets Authorization and Content-Type. Don’t put those in default_headers or you may duplicate them.

Step 9: Harden for production

Set timeouts and retries at construction. Read secrets from environment.

import os

llm = ChatOpenAI(
    model=os.getenv("GW_MODEL", "openai/gpt-4o"),
    base_url=os.getenv("GW_BASE_URL"),
    api_key=os.getenv("GW_API_KEY"),
    max_retries=3,
    timeout=30,
    temperature=0.1,
)

Add a LangChain callback to ship token counts to your own metrics:

from langchain_core.callbacks import StreamingStdOutCallbackHandler

llm = ChatOpenAI(
    base_url=os.getenv("GW_BASE_URL"),
    api_key=os.getenv("GW_API_KEY"),
    model="openai/gpt-4o",
    streaming=True,
    callbacks=[StreamingStdOutCallbackHandler()],
)

If the gateway provides automatic fallback, you can drop your own retry loop. If it doesn’t, max_retries on the client is your only shield against transient 529s.

Step 10: Switch models without code changes

Because the gateway abstracts providers, you can flip model at runtime:

def ask(prompt_text, model="anthropic/claude-3.5-sonnet"):
    m = ChatOpenAI(base_url=os.getenv("GW_BASE_URL"),
                   api_key=os.getenv("GW_API_KEY"), model=model)
    return m.invoke(prompt_text).content

This is the real win of a LangChain ChatOpenAI gateway base URL: one client, many backends, zero SDK swaps.

Verify success one more time

Run the Step 5 chain, confirm token usage prints, and check your gateway’s request log shows the expected model and route header. If the dashboard shows per-token metering and the right provider, you’re done.