Migrating from Vertex AI to a Gemini 3 gateway API

If you’re running Gemini workloads on Google Cloud and want to drop the Vertex SDK dependency, you can migrate Vertex AI to gateway API in a single afternoon. A gateway that exposes Gemini 3 through an OpenAI-compatible REST surface lets you keep prompt logic intact while removing project, region, and Google-auth coupling.

Step 1: Inventory your Vertex AI integration

Most production Vertex AI generative code looks like this:

from vertexai.generative_models import GenerativeModel, GenerationConfig

model = GenerativeModel("gemini-3-pro")
config = GenerationConfig(
    temperature=0.2,
    max_output_tokens=1024,
    top_p=0.95,
)
response = model.generate_content(
    "Summarize the Q3 infra report.",
    generation_config=config,
    stream=False,
)
print(response.text)

Extract four things before touching anything: model identifier, generation parameters, system instructions, and whether you stream. Vertex requires google.cloud.aiplatform credentials and a location such as us-central1. Note any use of system_instruction, safety_settings, or context caching via vertexai.preview.generative_models.CachedContent.

If you use streaming today, the Vertex loop looks like:

for chunk in model.generate_content("Stream a log analysis", stream=True):
    print(chunk.text, end="")

Capture the exact finish reasons and candidate counts. Chat Completions exposes only one candidate by default, so multi-candidate Vertex calls need redesign.

Step 2: Provision gateway credentials

export GATEWAY_BASE_URL="https://api.gateway.example/v1"
export GATEWAY_API_KEY="sk-..."

The gateway consolidates 240+ models behind one endpoint, so you only need to target the Gemini 3 model string. No Google Cloud project, service account JSON, or region configuration is required. Store the key in your secret manager exactly as you would an OpenAI key.

Step 3: Swap the client

Replace the Vertex SDK with the OpenAI Python client pointed at the gateway:

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["GATEWAY_BASE_URL"],
    api_key=os.environ["GATEWAY_API_KEY"],
)

That’s the only import change. The OpenAI client speaks Chat Completions, which the gateway translates to Gemini’s native API. If you previously instantiated a Gapic client with project and location, delete that block entirely.

Auth differences

Vertex used GOOGLE_APPLICATION_CREDENTIALS and OAuth scopes. The gateway uses a bearer token. Rotate the secret in your CI and remove the GCP ADC file from containers to shrink the attack surface.

Step 4: Translate model IDs and parameters

Vertex references models by short name inside a pinned location. The gateway uses a provider-prefixed slug that is stable across regions.

Vertex	Gateway
`gemini-3-pro` (in `us-central1`)	`google/gemini-3-pro`
`temperature`	`temperature`
`max_output_tokens`	`max_tokens`
`top_p`	`top_p`
`system_instruction`	`messages` with `role: "system"`
`candidate_count`	`n` (often ignored by Gemini)

Rewrite the call:

resp = client.chat.completions.create(
    model="google/gemini-3-pro",
    messages=[
        {"role": "system", "content": "You are a terse infra analyst."},
        {"role": "user", "content": "Summarize the Q3 infra report."},
    ],
    temperature=0.2,
    max_tokens=1024,
    top_p=0.95,
)
print(resp.choices[0].message.content)

If you used candidate_count > 1 in Vertex, drop it—Chat Completions returns one choice unless you set n, which Gemini 3 may ignore. Map stop_sequences to stop as a list of strings.

Step 5: Adapt streaming and response parsing

Vertex streaming yields GenerateContentResponse chunks with .text deltas. The gateway streams SSE tokens:

stream = client.chat.completions.create(
    model="google/gemini-3-pro",
    messages=[{"role": "user", "content": "Draft a runbook for etcd recovery."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="")

Map Vertex’s response.candidates[0].finish_reason to chunk.choices[0].finish_reason (stop, length, safety). Non-streaming responses nest content in resp.choices[0].message.content, not .text.

Tool calls

If you used Vertex function calling via tool_config, convert to OpenAI tools and tool_choice. Gemini 3 supports parallel calls; the gateway forwards them as tool_calls arrays on the assistant message.

Step 6: Handle context caching and cache-control

Vertex has explicit CachedContent objects with TTLs. Gateways forward provider cache-control hints through request extensions. If your gateway supports it, set a header or extra body field:

resp = client.chat.completions.create(
    model="google/gemini-3-pro",
    messages=[{"role": "user", "content": BIG_CONTEXT + " Explain the bottleneck."}],
    extra_headers={"x-provider-cache": "true"},
)

A gateway like n4n.ai honors client routing directives and forwards provider cache-control hints, so the same cached context works without Vertex’s SDK ceremony. Without the header, the gateway may still use provider-side caching if Gemini 3 supports prefix caching, but you lose explicit control.

Step 7: Add routing and fallback (optional)

If you want to avoid a single-provider lock-in, pass a routing directive. The gateway can pin to Google or fall back automatically when Vertex is degraded:

resp = client.chat.completions.create(
    model="google/gemini-3-pro",
    messages=[{"role": "user", "content": "Query the billing anomaly."}],
    extra_headers={"x-route": "google;fallback=azure"},
)

Automatic fallback when a provider is rate-limited or degraded keeps p99 latency stable after you migrate Vertex AI to gateway API. You can also set x-route: google;region=us-east if the gateway exposes region pins.

Step 8: Verify the migration

Write a parity test. Send the same prompt to both old and new paths (or stub Vertex) and assert structural equality:

def test_gateway_output():
    r = client.chat.completions.create(
        model="google/gemini-3-pro",
        messages=[{"role": "user", "content": "Return JSON: {'ok': true}"}],
        response_format={"type": "json_object"},
    )
    assert "ok" in r.choices[0].message.content
    assert r.usage.prompt_tokens > 0
    assert r.usage.completion_tokens > 0

Run it. Then check per-token usage metering in the gateway dashboard to confirm tokens are counted. If you see prompt_tokens and completion_tokens populated, the wire format is correct.

Curl sanity check:

curl $GATEWAY_BASE_URL/chat/completions \
  -H "Authorization: Bearer $GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"google/gemini-3-pro","messages":[{"role":"user","content":"ping"}]}'

A 200 with choices and usage means the migrate Vertex AI to gateway API cutover succeeded. Keep the Vertex route behind a feature flag for one release cycle in case Gemini 3 behavior drifts on edge cases.

Pitfalls to avoid

Safety filters: Vertex returns safety_ratings per candidate. Gateways may omit them or surface as extensions. Don’t assume they exist in the response.
System instructions in Vertex are separate from chat history. In OpenAI format they must be the first message or they get ignored.
Region latency: Vertex colocates with your GCP resources. A gateway adds a TLS termination hop; measure p95 before decommissioning the old path.
Fine-tuned models: Vertex model IDs like gemini-3-pro-002 map to gateway slugs differently. Check the gateway’s model list.
Streaming timeouts: Vertex SDK handles retry on broken streams. OpenAI client does too, but tune max_retries if you see truncated outputs.

Final checklist

Vertex SDK removed from requirements.
All generate_content calls replaced with chat.completions.create.
Model strings use google/gemini-3-pro style.
Streaming loops use .delta.content.
Cache headers set where Vertex used CachedContent.
Usage metering verified.

Following these steps lets you migrate Vertex AI to gateway API without rewriting prompt engineering or eval harnesses. Gemini 3 behaves the same; only the transport changed.