One API, every model: routing GPT-5 and Gemini in LangGraph

Most LangGraph agents hardcode a single LLM, then break when you need to route GPT-5 and Gemini in LangGraph for cost, latency, or capability reasons. This guide gives an ordered path to build a conditional router that picks the right backend per task, and shows how a unified inference gateway can collapse the provider gap entirely.

Prerequisites

Install the current LangChain model bindings and LangGraph:

pip install langgraph langchain-openai langchain-google-genai

Set provider credentials as environment variables (OPENAI_API_KEY, GOOGLE_API_KEY). If you use a gateway, only its key is required.

1. Instantiate provider-specific clients

LangGraph does not care which BaseChatModel you pass to a node, but the two providers ship different SDK wrappers. Create both clients up front so the graph can switch without reinitialization cost.

from langchain_openai import ChatOpenAI
from langchain_google_genai import ChatGoogleGenerativeAI

gpt5 = ChatOpenAI(model="gpt-5", temperature=0)
gemini = ChatGoogleGenerativeAI(model="gemini-1.5-pro", temperature=0)

Keep the instances module-level. Reconstructing the client per invocation adds 10–50 ms of dead time and defeats connection pooling.

2. Define routing criteria

Routing should be explicit, not a coin flip. A common split: GPT-5 for structured code generation and tool calling, Gemini for long-context summarization or multimodal ingestion. Start with a deterministic heuristic; escalate to an LLM classifier only if the heuristic misses.

def classify_task(text: str) -> str:
    lowered = text.lower()
    if any(k in lowered for k in ("summarize", "extract", "long doc")):
        return "summarize"
    return "generate"

If you need semantic routing, call a small model inside classify and return its label. That adds a round trip, so cache the result on the state.

3. Model the LangGraph state and nodes

Use a typed state with the message list and the routing decision. Nodes are plain functions that take the state and return a partial update.

from typing import TypedDict, List
from langchain_core.messages import HumanMessage, AIMessage

class AgentState(TypedDict):
    messages: List[HumanMessage | AIMessage]
    task_type: str

def classify_node(state: AgentState) -> dict:
    last = state["messages"][-1].content
    return {"task_type": classify_task(last)}

def gpt5_node(state: AgentState) -> dict:
    resp = gpt5.invoke(state["messages"])
    return {"messages": [resp]}

def gemini_node(state: AgentState) -> dict:
    resp = gemini.invoke(state["messages"])
    return {"messages": [resp]}

Note that invoke returns an AIMessage. Storing it directly keeps the graph reducible—LangGraph appends or replaces based on your reducer, so define messages with Annotated[list, add_messages] if you want auto-append.

4. Wire the conditional edge

The graph entry point classifies, then a conditional edge sends control to the matching model node. Both nodes terminate the run.

from langgraph.graph import StateGraph, END
from langchain_core.messages import add_messages
from typing import Annotated

class AgentState(TypedDict):
    messages: Annotated[list, add_messages]
    task_type: str

sg = StateGraph(AgentState)
sg.add_node("classify", classify_node)
sg.add_node("gpt5", gpt5_node)
sg.add_node("gemini", gemini_node)

sg.set_entry_point("classify")
sg.add_conditional_edges(
    "classify",
    lambda s: s["task_type"],
    {"generate": "gpt5", "summarize": "gemini"}
)
sg.add_edge("gpt5", END)
sg.add_edge("gemini", END)

app = sg.compile()

Run it:

result = app.invoke({"messages": [HumanMessage(content="Summarize this 200-page PDF")]})
print(result["messages"][-1].content)

The router is now a first-class graph edge, so you can swap in a third provider by adding one node and one dict entry.

5. Collapse providers with one endpoint

Maintaining two SDKs means two error taxonomies, two retry policies, and divergent tool-call schemas. An OpenAI-compatible inference gateway removes that friction: you point ChatOpenAI at a single base URL and change only the model string.

from langchain_openai import ChatOpenAI

def get_model(model_name: str) -> ChatOpenAI:
    return ChatOpenAI(
        model=model_name,
        base_url="https://api.n4n.ai/v1",
        api_key="YOUR_GATEWAY_KEY",
        temperature=0,
    )

gpt5 = get_model("gpt-5")
gemini = get_model("gemini-1.5-pro")

n4n.ai provides one OpenAI-compatible endpoint that addresses 240+ models and automatic fallback when a provider is rate-limited; it also honors client routing directives and forwards provider cache-control hints, so the same get_model call can pin a region or request cache retention without SDK changes. Your LangGraph nodes stay identical—they only depend on the BaseChatModel interface.

Tradeoff: you lose provider-specific knobs (e.g., Gemini’s safety_settings) unless the gateway exposes them as passthrough headers. For most routing use cases the uniformity wins.

6. Common pitfalls and tradeoffs

Context window mismatch. Gemini 1.5 Pro handles up to 1M tokens; GPT-5’s window is smaller. If you route a 300k-token doc to GPT-5, the client will raise before the graph can recover. Truncate or pre-route by length:

def classify_task(text: str) -> str:
    if len(text) > 100_000:
        return "summarize"  # gemini path
    ...

Tool call schema drift. LangChain normalizes tool calls, but Gemini and OpenAI serialize arguments differently. Test every tool against both models before trusting the router in production.

Latency from the classifier. A heuristic is sub-millisecond. An LLM classifier adds a full inference call. If your graph already calls a model, fold the routing decision into the system prompt and parse the first token instead of a separate node.

Error handling and fallback. Provider outages are not theoretical. Wrap each node in a retry with a fallback to the other model:

def gpt5_node(state):
    try:
        return {"messages": [gpt5.invoke(state["messages"])]}
    except Exception:
        return {"messages": [gemini.invoke(state["messages"])]}

If you use a gateway with automatic fallback, this logic moves out of your code.

State pollution. Messages returned by different providers carry different response_metadata. When persisting state to a database, serialize only content and tool_calls to avoid bloat.

Cost metering. Per-token cost varies wildly between GPT-5 and Gemini. If you bill per request, capture response_metadata["usage"] from each node and emit it to your metering pipeline. Gateways that return standardized usage simplify this.

7. Production checklist

• Pin model versions (gpt-5, gemini-1.5-pro) not aliases that drift.
• Set timeouts on the clients (.with_config({"timeout": 30})).
• Log the task_type decision for every run to audit routing quality.
• Load-test the graph with both providers under your real payload sizes.
• If using a unified endpoint, verify it forwards cache-control so repeated long-context calls hit provider prefix caches.

Routing GPT-5 and Gemini in LangGraph is fundamentally a state-machine problem, not a model problem. Get the edges right, keep the nodes thin, and the swap from one backend to another becomes a one-line config change.