Switching LangChain from OpenAI to Claude 3.5 via a gateway

The langchain openai to claude 3.5 switch is mostly a configuration change if your stack already calls OpenAI’s chat completions API through LangChain. By routing ChatOpenAI at an OpenAI-compatible gateway, you swap the base URL and model identifier instead of rewriting chains, prompts, or tool schemas.

Step 1: Inventory your existing LangChain OpenAI calls

Find every place you instantiate an OpenAI chat model. In modern LangChain (langchain-openai package), that’s ChatOpenAI.

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-4o",
    temperature=0.2,
    max_tokens=1024,
)

Grep your codebase for ChatOpenAI, OpenAI, and AzureChatOpenAI. Note the model strings, any openai_api_base overrides, and whether you rely on streaming, functions/tools, or response_format. The langchain openai to claude 3.5 switch only gets risky when you depend on OpenAI-specific response shapes (e.g., JSON mode via response_format={"type": "json_object"}), because Claude handles structured output differently.

Step 2: Point LangChain at the gateway instead of OpenAI

A gateway like n4n.ai exposes one OpenAI-compatible endpoint for 240+ models, so you keep the same client class. Set the base URL and API key via environment variables or constructor arguments.

export OPENAI_API_BASE="https://gateway.example.com/v1"
export OPENAI_API_KEY="sk-gateway-..."

import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="anthropic/claude-3.5-sonnet",
    openai_api_base=os.getenv("OPENAI_API_BASE"),
    openai_api_key=os.getenv("OPENAI_API_KEY"),
    temperature=0.2,
    max_tokens=1024,
)

If you previously used openai_api_base for Azure, remove the Azure-specific versioning query params; a standard gateway speaks the vanilla /v1/chat/completions route.

Step 3: Map model names and parameter defaults

OpenAI model names do not exist on the gateway’s Claude side. Use the gateway’s model ID for Claude 3.5 Sonnet (commonly anthropic/claude-3.5-sonnet or claude-3.5-sonnet depending on the provider prefix).

OpenAI model	Gateway Claude 3.5 ID
`gpt-4o`	`anthropic/claude-3.5-sonnet`
`gpt-4-turbo`	`anthropic/claude-3.5-sonnet`

Claude 3.5 requires max_tokens on every request. LangChain’s ChatOpenAI forwards max_tokens correctly, but if your old code omitted it (relying on OpenAI’s default 4096), set it explicitly to avoid a 400 from the upstream provider.

Temperature scales similarly, but Claude’s top-p is not exposed via the OpenAI route—drop top_p overrides unless the gateway documents a mapping.

Step 4: Adapt message construction and system prompts

Claude expects a top-level system field, while OpenAI mixes system messages into the array. A compliant gateway translates role: "system" messages automatically, so your existing LangChain SystemMessage keeps working.

from langchain_core.messages import SystemMessage, HumanMessage

messages = [
    SystemMessage(content="You are a terse API assistant."),
    HumanMessage(content="Summarize the LangChain switch in one line."),
]

resp = llm.invoke(messages)
print(resp.content)

If you previously injected system instructions via a leading HumanMessage, move them to SystemMessage for cleaner Claude behavior. The langchain openai to claude 3.5 switch is a good moment to audit prompt placement.

Step 5: Handle streaming, tools, and callbacks

Streaming works unchanged: pass streaming=True and consume llm.stream(messages).

for chunk in llm.stream(messages):
    print(chunk.content, end="", flush=True)

Tool calling uses LangChain’s bind_tools. The gateway converts OpenAI-style tools arrays into Claude’s tool format.

from langchain_core.tools import tool

@tool
def get_weather(city: str) -> str:
    """Fetch weather for a city."""
    return f"Sunny in {city}"

llm_with_tools = llm.bind_tools([get_weather])
ai_msg = llm_with_tools.invoke("What's the weather in Berlin?")
print(ai_msg.tool_calls)

If you used OpenAI’s function_call forcing, replace it with tool_choice dicts—Claude 3.5 supports equivalent selective invocation through the gateway.

Step 6: Run a smoke test and verify success

Write a minimal script that exercises your real message shape and prints token usage.

import os
from langchain_openai import ChatOpenAI
from langchain_core.messages import SystemMessage, HumanMessage

llm = ChatOpenAI(
    model="anthropic/claude-3.5-sonnet",
    openai_api_base=os.getenv("OPENAI_API_BASE"),
    openai_api_key=os.getenv("OPENAI_API_KEY"),
    max_tokens=256,
)

out = llm.invoke([
    SystemMessage(content="Reply with 'ok' only."),
    HumanMessage(content="Ping"),
])

print("Content:", out.content)
print("Usage:", out.usage_metadata)
print("Model:", out.response_metadata.get("model"))

Success criteria:

out.content returns coherent text (not an OpenAI error).
out.usage_metadata shows input_tokens and output_tokens (LangChain normalizes Claude’s usage).
response_metadata.model reflects the Claude 3.5 identifier, confirming the langchain openai to claude 3.5 switch routed correctly.

If you see 401, your gateway key is wrong. 400 with max_tokens missing means you skipped Step 3. 422 on tools means the gateway rejected a schema field Claude doesn’t support (e.g., strict mode).

Step 7: Lock the switch in tests and CI

Replace hardcoded model names with a config value. In pytest, mock the gateway with respx or pytest-httpx and assert the outbound JSON contains "model": "anthropic/claude-3.5-sonnet".

def test_model_routing(respx_mock):
    respx_mock.post("https://gateway.example.com/v1/chat/completions").mock(
        return_value=httpx.Response(200, json={"choices": [{"message": {"content": "ok"}}], "usage": {}})
    )
    # assert llm.model == "anthropic/claude-3.5-sonnet"

Cache the gateway base URL in your deployment secrets. The langchain openai to claude 3.5 switch is complete once CI passes against the gateway and your production logs show Claude 3.5 responses with expected latency.

Verification checklist

All ChatOpenAI instances use the gateway base URL.
max_tokens set on every call.
System prompts in SystemMessage, not buried in history.
Tool schemas validated against Claude’s parameter limits.
Smoke test prints usage metadata with non-zero token counts.

Following these steps, the langchain openai to claude 3.5 switch takes an afternoon, not a rewrite. Your chains stay intact; only the transport and model label change.