Migrating a production app from GPT-4 to Gemini 2.5

Most teams treat a GPT-4 to Gemini 2.5 migration as a risky rewrite. It isn’t. The work is concentrated in three places: request shaping, tool-call decoding, and streaming semantics. If you audit those, the swap is mechanical.

Prerequisites

• Python 3.11+ with openai and google-genai installed (pip install openai google-genai).
• Existing production code that calls GPT-4 via the OpenAI SDK (model gpt-4-turbo or gpt-4o).
• A Gemini API key (or a gateway key if using an OpenAI-compatible proxy).
• A corpus of recorded production prompts and tool traces for regression testing.
• Familiarity with your LLM spend metering; you will need to verify token counts post-migration.

Step 1: Inventory your GPT-4 call sites

Run a static scan for client.chat.completions.create. Record the parameters you actually use: model, temperature, max_tokens, tools, response_format, stream. Guessing from memory wastes time.

# audit.py
import ast, pathlib

calls = []
for p in pathlib.Path("app").rglob("*.py"):
    tree = ast.parse(p.read_text())
    for node in ast.walk(tree):
        if isinstance(node, ast.Call) and getattr(node.func, "attr", "") == "create":
            calls.append(node)
print(len(calls), "call sites")

Expected output: 37 call sites (your number will vary). Focus on whether you rely on tool_calls or response_format={"type": "json_object"}. Gemini 2.5 supports function calling and JSON mode, but the wire format and config keys differ. Note every place you parse finish_reason or usage—those fields shift shape.

Step 2: Choose an integration path

You have two options: call Gemini natively with google-genai, or keep the OpenAI client and point it at a compatible gateway. The second minimizes code churn and preserves your existing observability hooks.

If you route through n4n.ai, the GPT-4 to Gemini 2.5 migration is a model string change because its OpenAI-compatible endpoint addresses 240+ models and forwards provider cache-control hints without code changes.

# openai_compatible.py
from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",  # OpenAI-compatible
    api_key="YOUR_KEY",
)

resp = client.chat.completions.create(
    model="gemini-2.5-pro",
    messages=[{"role": "user", "content": "Summarize: ..."}],
    max_tokens=512,
)
print(resp.choices[0].message.content)

Native SDK requires restructuring messages into Gemini’s contents and system_instruction shape, which forces a refactor of your request builder.

# native.py
from google import genai

client = genai.Client(api_key="GEMINI_KEY")
resp = client.models.generate_content(
    model="gemini-2.5-pro",
    contents="Summarize: ...",
    config={"system_instruction": "You are a terse editor.", "max_output_tokens": 512},
)
print(resp.text)

Step 3: Map system prompts and message history

GPT-4 takes a system role message. Gemini 2.5 separates system instruction from user/model turns. Write a pure transformer so you can unit-test it.

def to_gemini(messages):
    sys = next((m["content"] for m in messages if m["role"] == "system"), None)
    turns = []
    for m in messages:
        if m["role"] == "system":
            continue
        role = "user" if m["role"] == "user" else "model"
        turns.append({"role": role, "parts": [m["content"]]})
    return sys, turns

Call it before the native client:

sys, turns = to_gemini(messages)
resp = client.models.generate_content(
    model="gemini-2.5-pro",
    contents=turns,
    config={"system_instruction": sys},
)

Checkpoint: feed a recorded prompt where GPT-4 returned a known summary. Expected native output prints the same intent as GPT-4, though phrasing and ordering differ. If sys is None, pass system_instruction=None explicitly—Gemini rejects empty strings.

Step 4: Port tool calls

OpenAI returns message.tool_calls with function.name and arguments as a JSON string. Gemini returns function_calls attached to response parts, with args already decoded to a dict.

# OpenAI style
if msg.tool_calls:
    for tc in msg.tool_calls:
        fn = tc.function.name
        args = json.loads(tc.function.arguments)

# Gemini style
if resp.function_calls:
    for fc in resp.function_calls:
        fn = fc.name
        args = fc.args  # already a dict

Wrap both behind an adapter so the rest of your app stays provider-agnostic:

def extract_tools(resp, provider):
    if provider == "openai":
        return [(tc.function.name, json.loads(tc.function.arguments))
                for tc in resp.choices[0].message.tool_calls]
    else:
        return [(fc.name, fc.args) for fc in resp.function_calls]

Gemini 2.5 supports parallel calls in one turn; OpenAI does too, but your executor must handle a list either way. One difference: Gemini may return a textual response alongside function_calls in the same turn. Your loop must not assume mutual exclusion.

Step 5: Handle streaming and finish reasons

GPT-4 streaming yields delta.content chunks; Gemini streams chunks with .text. Finish reason length maps to Gemini’s MAX_TOKENS, but Gemini also emits STOP and SAFETY reasons you likely ignored before.

# OpenAI stream
for chunk in openai_stream:
    if chunk.choices[0].delta.content:
        yield chunk.choices[0].delta.content

# Gemini stream
for chunk in client.models.generate_content_stream(
    model="gemini-2.5-pro", contents=turns, config=cfg
):
    if chunk.text:
        yield chunk.text

If you need to detect truncation, check chunk.finish_reason == "MAX_TOKENS" in Gemini vs chunk.choices[0].finish_reason == "length" in OpenAI. Log both so your token-limit alerts keep working.

Step 6: Validate with recorded traffic

Replay 200 logged conversations. Compare outputs with a cheap heuristic: embedding cosine similarity > 0.85 and tool-name match rate = 100%. Do not trust a single golden string.

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

def parity(old, new, emb):
    a, b = emb(old), emb(new)
    return cosine_similarity([a], [b])[0][0]

# print mean score
print(np.mean([parity(o, n, embed) for o, n in pairs]))

Expected output: 0.88 (your corpus will differ; treat <0.8 as review-required). Tool-name mismatch is a hard fail regardless of similarity—it means your schema mapping broke.

Step 7: Ship behind a flag with fallback

Put the model choice behind a config toggle. If you use a gateway that honors client routing directives, you can request Gemini but automatically fall back to GPT-4 when Gemini is degraded or rate-limited.

client.chat.completions.create(
    model="gemini-2.5-pro",
    messages=messages,
    extra_headers={"x-fallback-allow": "gpt-4o"},  # gateway-specific
)

Without a gateway, implement a try/except in your caller:

try:
    return call_gemini(messages)
except GeminiError as e:
    log.warning("gemini failed: %s", e)
    return call_gpt4(messages)

Monitor token usage per provider. Gemini 2.5 counts cached input tokens differently; verify your metering matches provider bills before turning off GPT-4 entirely.

Gotchas we hit in production

• Gemini 2.5 rejects empty system_instruction; pass None not "".
• Function argument schemas must use type as string, not Python int—same as OpenAI, but Gemini is stricter on additionalProperties.
• Streaming latency for first token is lower on Gemini for long contexts, but mid-stream jitter is higher; set a 10s idle timeout.
• JSON mode in Gemini uses response_mime_type="application/json" in config, not a separate response_format object.
• Gemini’s max_output_tokens is inclusive of thinking tokens for reasoning models; set it higher than your old max_tokens if you used gpt-4o without reasoning overhead.

Final checklist

• All create call sites route through adapter.
• System prompts mapped to system_instruction.
• Tool call extraction unified and parallel calls handled.
• Streaming finish reasons mapped and logged.
• Parity replay >0.8 mean similarity, tool names 100% match.
• Fallback path tested by injecting errors in chaos run.

A GPT-4 to Gemini 2.5 migration is finished when your adapter tests pass and the fallback fires correctly in chaos tests. Then flip the flag and watch the usage dashboard for a week.