Comparing vision model support across LLM API gateways

Building multimodal features means evaluating vision model support LLM API gateways before you commit to a provider. The differences in model coverage, image handling, and cost structure are larger than most teams expect, and the wrong choice can lock you into a single vendor’s quirks.

The contenders

We compare four ways to access vision-capable models:

• OpenAI API – first-party endpoint for GPT-4o, GPT-4 Turbo, and older vision models.
• OpenRouter – aggregator that routes to 300+ models including vision variants from OpenAI, Anthropic, Google, and open-weight providers.
• Azure OpenAI Service – enterprise gateway reselling OpenAI models with private networking and compliance controls.
• Vertex AI (Gemini) – Google Cloud’s first-party API for Gemini 1.5 Pro/Flash and related multimodal models.

Each exposes vision input differently, but all accept base64 or URL images in some form.

Capabilities

When comparing vision model support LLM API gateways, the first gap is modality coverage. Vision is not monolithic; the core question is what formats the gateway passes through unchanged.

OpenAI’s API accepts one or more images (PNG, JPEG, WEBP, non-animated GIF) per message, downsampled to 512×512 tiles for token counting. It does not natively ingest PDFs unless you use the file API beta. Video is out of scope for the chat endpoint.

Claude models behind OpenRouter accept up to 20 images per message, with a hard pixel limit around 8000×8000. They handle PDF pages as images natively, which is useful for document workflows.

Gemini on Vertex AI takes images, PDFs, and short video clips in a single content block, with native token accounting per modality. It supports multiple images and audio in the same turn, making it the only first-party option here with video input.

Aggregators like OpenRouter forward whatever the upstream provider accepts. They do not normalize image schemas; you still must match the underlying model’s expectations. If you send a PDF to a GPT-4o route, it will fail unless the model supports it.

A subtle capability gap is cache control. Providers like Anthropic expose cache_control on image blocks. Gateways that honor client routing directives and forward provider cache-control hints preserve this; others strip it.

Price and cost model

The cost of vision model support LLM API gateways splits between first-party and aggregator models. First-party APIs price vision by token, but image tokens are computed via resolution tiling. OpenAI charges per 512×512 tile for high-res; GPT-4o mini is cheaper. Gemini uses a per-image flat token estimate that scales with file size. Claude counts image tokens by pixel buckets.

OpenRouter publishes provider cost plus a fixed markup; you pay per routed token with no commitment. Azure OpenAI inherits OpenAI pricing but adds enterprise support fees and requires capacity reservations for high throughput. Vertex AI Gemini pricing is per modality with no separate gateway margin.

None of the gateways offer free vision quotas beyond provider free tiers (e.g., Gemini free tier limits).

Latency and throughput

First-party endpoints give the lowest median latency because there is no proxy hop. OpenAI’s vision latency for a single 1024×1024 image on GPT-4o is typically 400–900 ms to first token, depending on region.

OpenRouter adds 20–100 ms of routing overhead. Its anycast DNS picks a region, but you share bandwidth with other tenants. Azure OpenAI can deliver lower p99 if you provision dedicated capacity, but default standard deployments share quota with other subscriptions in the region.

Vertex AI Gemini latency is competitive with OpenAI, but cold starts on non-default regions can spike. Aggregators without fallback will surface provider 429s directly; some gateways retry on another provider transparently, which reduces tail latency.

Ergonomics

OpenAI-compatible schemas dominate. OpenAI, Azure OpenAI, and (with a model prefix) OpenRouter accept the exact same messages payload with type: "image_url". Migration between them is a base-URL swap.

from openai import OpenAI
client = OpenAI(base_url="https://openrouter.ai/api/v1", api_key="...")
resp = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this diagram"},
            {"type": "image_url", "image_url": {"url": "https://example.com/img.png"}}
        ]
    }]
)

Vertex AI uses a different contents array with inline_data blobs; you need the Google SDK or a translation layer. Streaming works everywhere. Function calling with images is supported on OpenAI, Gemini, and Claude, but aggregators only pass through the capability if the model declares it.

SDK support: OpenAI and Azure have official Python/TS clients. Vertex has Google Cloud SDK. OpenRouter is drop-in with the OpenAI client if you set base_url.

Ecosystem and tooling

OpenAI has the richest tooling: assistants, limited fine-tuning of vision models, and a large community. Azure adds Entra ID, private endpoints, and compliance certifications. Vertex bundles with BigQuery, Dialogflow, and Google’s MLOps.

OpenRouter’s value is model breadth—you can A/B test Claude Opus against GPT-4o in one loop without managing multiple keys.

Limits and quotas

Concrete limits engineers hit:

• OpenAI: 20 images per request on GPT-4o; max 5MB per image; rate limits by tier (e.g., 10k RPM on tier 4).
• Azure OpenAI: same model limits, but subscription quotas are manually adjustable.
• Vertex AI: 16 images per prompt on Gemini 1.5 Pro; file size caps at 20MB for inline.
• OpenRouter: per-model limits inherited; global spend caps configurable.

Image resolution beyond 2000px is auto-downscaled by most gateways, losing detail.

Comparison table

Which to choose

Prototype fast, single model: Use OpenAI API directly. Best docs, lowest latency, no proxy confusion.

Experiment across many vision models: OpenRouter. One endpoint for 300+ models, easy A/B testing without multi-key code.

Regulated enterprise, data residency: Azure OpenAI. Same vision models with private networking and audits.

Google-cloud-native, PDF/video heavy: Vertex AI Gemini. Native multimodal ingestion saves preprocessing.

High-volume production with multi-provider resilience: Azure with reserved capacity, or OpenRouter if you want breadth without enterprise contracts.

Pick based on where your images live and how much routing control you need. The vision model support LLM API gateways offer is converging on OpenAI schema, but the limits and failover behavior still differ enough to matter.