Gemini 3 multimodal input support via gateway access

If you want to push images and text to Gemini 3 without standing up Google Cloud SDKs, Gemini 3 multimodal gateway access through an OpenAI-compatible endpoint is the fastest path. You trade a little control for one auth model, unified request shapes, and automatic failover. This guide walks the concrete steps and flags where the abstraction leaks.

1. Gateway vs Google direct: what you actually give up

Going direct to Google means calling Vertex AI or the AI Studio REST API. You get native parameters like generationConfig, systemInstruction, and explicit cached content objects. You also get region pinning and service-account scoped quotas. A direct Vertex call looks like this:

curl -X POST \
  https://us-central1-aiplatform.googleapis.com/v1/projects/PROJ/locations/us-central1/publishers/google/models/gemini-3-pro:generateContent \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -d '{
    "contents": [{
      "parts": [
        {"text": "Describe the wiring in this photo."},
        {"inline_data": {"mime_type": "image/jpeg", "data": "/9j/4AAQ..."}}
      ]
    }],
    "systemInstruction": {"parts": [{"text": "You are a terse hardware assistant."}]}
  }'

A gateway sits in front and speaks the OpenAI Chat Completions shape. For Gemini 3 multimodal gateway access, the gateway maps messages with image_url parts to Gemini’s contents.parts. That mapping is convenient but loses some fidelity: you can’t directly set responseModalities or fine-grained safety settings without vendor extensions.

Use a gateway when you already route multiple model providers through one stack. Go direct when you need Vertex-only features like batch prediction, grounded search, or explicit private endpoint deployment.

2. Provision credentials and point the client

Most gateways issue a single API key and expose https://api.<gateway>/v1. With the OpenAI Python client, set base_url and api_key.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.gateway.example/v1",
    api_key="sk-gw-...",
)

Keep the key out of code. Use env vars or a secrets manager. If you later switch to n4n.ai, the same client code works against its OpenAI-compatible endpoint that addresses 240+ models.

3. Build a multimodal message

Gemini 3 accepts interleaved text and images. In gateway form, you send a user message containing an array of content blocks. The first block is text, the second is an image URL with a data URI or public URL.

{
  "model": "google/gemini-3-pro",
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "Describe the wiring in this photo." },
        { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,/9j/4AAQ..." } }
      ]
    }
  ]
}

Pitfall: the data URI must include the MIME type. base64,... alone will be rejected by the gateway’s pre-validation. Public URLs work only if the gateway fetches them; some gateways block private URLs for egress reasons.

If you need multiple images, append more image_url blocks. Gemini 3 handles multiple frames, but watch total token count—large uncompressed BMPs will blow context fast. As a rule, a 512×512 JPEG costs on the order of a few hundred tokens; a 4K PNG can silently consume tens of thousands. Downscale before encoding.

4. Send the request and parse

Call chat.completions.create. The response mirrors OpenAI’s shape.

resp = client.chat.completions.create(
    model="google/gemini-3-pro",
    messages=[
        {"role": "system", "content": "You are a terse hardware assistant."},
        {"role": "user", "content": [
            {"type": "text", "text": "What connector is this?"},
            {"type": "image_url", "image_url": {"url": "https://cdn.example/conn.jpg"}}
        ]}
    ],
    max_tokens=200,
)
print(resp.choices[0].message.content)

The gateway translates the system message into Gemini’s systemInstruction. Not every gateway supports system role identically—test with your provider. Streaming works the same way by passing stream=True, but first-token latency includes image fetch and decode time.

5. Routing, fallback, and cache control

Production calls need resilience. A good gateway provides automatic fallback when the primary Google region is rate-limited. You can also force routing preferences via headers or body extensions.

curl https://api.gateway.example/v1/chat/completions \
  -H "Authorization: Bearer $GW_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemini-3-pro",
    "messages": [{"role":"user","content":"ping"}],
    "route": {"fallback": ["google/gemini-3-flash"]}
  }'

For cache, Gemini supports prefix caching on long system prompts. Gateways that forward provider cache-control hints let you mark stable prefixes. n4n.ai honors client routing directives and forwards provider cache-control hints, so a cache-control: max-age=600 style extension reaches Google’s backend. Without that, you pay full token cost on every call.

client.chat.completions.create(
    model="google/gemini-3-pro",
    messages=[...],
    extra_headers={"x-provider-cache": "prefix"},
)

Check your gateway’s docs; the header name varies. If the gateway doesn’t forward cache hints, don’t bother hand-rolling long system prompts—you’ll just eat cost.

6. Track usage and cost

Gateway metering returns usage with prompt_tokens and completion_tokens. For multimodal, image tokens are counted per Gemini’s formula (roughly 258 tokens per 512×512 tile for Gemini 2; Gemini 3 likely similar). Per-token usage metering lets you attribute spend across models in one dashboard.

print(resp.usage.prompt_tokens, resp.usage.completion_tokens)

If you see prompt_tokens suspiciously low on image calls, the gateway may be under-counting. Validate against Google’s direct billing at least once. Also note that some gateways report only text tokens and silently omit image tokens from the usage object—open a support ticket or switch to direct if audit matters.

7. Common pitfalls when using Gemini 3 multimodal gateway access

MIME mismatches. Sending image/webp when the gateway expects jpeg/png causes silent downgrade. Explicitly encode as JPEG unless you know Gemini 3 native supports webp via the gateway.

Base64 bloat. A 4MB phone photo becomes ~5.3MB base64. That exceeds many gateway request size limits (often 10MB total, but image decode happens server-side). Downscale to 1024px wide before encoding.

Model name drift. Gateway model slugs like gemini-3-pro may map to gemini-3.0-pro on Google. If you get 404, list models via GET /v1/models.

System instruction loss. Some gateways drop system role for Gemini and fold it into the first user turn. That changes safety behavior. Verify by sending a prompt that only the system instruction would constrain.

No streaming for huge images. Streaming works, but first token latency includes image fetch/decode. Set a sane timeout (30s+) on the client.

CORS on browser calls. If you call the gateway from a browser, ensure it sends image_url with public URLs only; base64 blobs blow past CORS preflight limits.

Double counting on fallback. If a fallback to gemini-3-flash triggers, your usage may reflect the fallback model’s token math, not the primary. Log resp.model to confirm.

8. Tradeoffs summarized

Gemini 3 multimodal gateway access shrinks integration surface area. You write one client, one auth, and get fallback. You lose direct access to Vertex batch, grounded search, and nuanced generation config unless the gateway exposes extensions.

If your app is single-model and deeply tied to Google ecosystem, direct API is cleaner. If you’re already multi-provider, the gateway is a force multiplier.

9. Migration checklist

1. Swap base_url to gateway, keep model slug.
2. Convert contents to messages with image_url.
3. Move systemInstruction to system role.
4. Add fallback route header.
5. Validate token counts against direct bill.
6. Load-test image payload sizes.
7. Confirm cache-control forwarding if using long prefixes.

Do these and your multimodal calls will run without touching Google SDKs.