Multimodal context windows compared across providers

A multimodal context window comparison across providers shows that image tokenization, max context, and API shapes diverge more than the marketing implies. OpenAI, Anthropic, and Google each handle vision and long context differently, and those differences directly affect cost and latency in production.

The contenders

We compare three first-party APIs that ship multimodal models with public endpoints:

• OpenAI – gpt-4o and gpt-4-turbo with vision, 128K context.
• Anthropic – claude-3-opus/sonnet/haiku with vision, 200K context.
• Google – gemini-1.5-pro with 1M–2M context, native image/audio/video.

Each exposes an HTTP API, but the wire format for images and the token accounting differ enough that you cannot swap them with a single adapter.

Capabilities: what fits in the window

OpenAI counts an image as a fixed token block based on resolution grid (e.g., low/medium/high). A 512x512 image at “high” is tiled and can consume ~700+ tokens. Anthropic computes tokens from the decoded image size and returns the count in the response; a 1MB PNG typically uses 1–2K tokens. Google embeds images as inline_data parts and counts them against the 1M window with no separate per-image cap.

Long context is where Google wins: Gemini 1.5 Pro handles 1M tokens (2M in preview) with interleaved text, images, and video frames. OpenAI caps at 128K total; Anthropic at 200K. If your use case is “PDF with 500 pages plus diagrams,” only Google fits it without chunking.

Price and cost model

No provider charges a flat monthly fee for API access; all meter by token. The wrinkle is that “token” for images is not the same unit as text.

• OpenAI bills image tokens at the same rate as text input tokens for gpt-4o, but the token count is determined by resolution preset.
• Anthropic bills input tokens (including image tokens) at the published per-Mtok price; vision does not add surcharge.
• Google bills Gemini 1.5 Pro at a single rate for the first 128K tokens and a higher rate beyond that, so a 1M-token multimodal prompt costs disproportionately more.

Do not assume a 200K context window is “free” to fill. The marginal cost of a 150K-token request with ten images is dominated by the text, not the pixels, on Anthropic; on Google it may cross the 128K pricing boundary.

Latency and throughput

Measured anecdotally: OpenAI returns first token for a small image+question in ~300–600ms. Anthropic similar. Google’s Gemini 1.5 Pro with a 800K-token context often takes multiple seconds before first token, regardless of image count, because it processes the full prefix.

Throughput scales inversely with context length on all three. If you need low latency on a long document, pre-cache the static prefix. Google supports cached_content; Anthropic supports prompt caching via cache_control markers; OpenAI supports seed+parallel calls but no server-side prefix cache yet.

Ergonomics

The request shapes are the pain point. OpenAI uses a content array with type: "image_url":

import openai
resp = openai.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What is in this diagram?"},
            {"type": "image_url", "image_url": {"url": "https://example.com/d.png"}}
        ]
    }]
)

Anthropic expects a content array with type: "image" and base64 source:

import anthropic
resp = anthropic.messages.create(
    model="claude-3-sonnet-20240229",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What is in this diagram?"},
            {"type": "image", "source": {
                "type": "base64",
                "media_type": "image/png",
                "data": "iVBORw0KGgo..."
            }}
        ]
    }]
)

Google uses parts with inline_data:

import google.generativeai as genai
model = genai.GenerativeModel("gemini-1.5-pro")
resp = model.generate_content([
    "What is in this diagram?",
    {"mime_type": "image/png", "data": b"..."}
])

None of these are wire-compatible. A gateway that presents an OpenAI-compatible endpoint can mask the differences; for example, n4n.ai honors client routing directives and forwards provider cache-control hints, so you can keep the OpenAI shape and target Claude or Gemini via header.

Ecosystem and tooling

OpenAI has the deepest ecosystem: function calling, JSON mode, and dozens of wrapper libraries. Anthropic offers tool use and solid Python/TS SDKs, but fewer third-party integrations. Google’s SDK is mature for Vertex, but the generative language API lags in community tooling.

If you rely on RAG frameworks like LangChain, all three are supported, but multimodal retrievers are still nascent. You will likely write the image chunking logic yourself.

Limits and gotchas

• OpenAI: max 10 images per message on some models; detail: low reduces tokens but drops resolution.
• Anthropic: max 20 images per request on Claude 3; image bytes must be base64, no URLs.
• Google: can ingest up to 3.5K images in a 1M context, but each generate_content call has a 20MB request size limit.

All three reject non-RGB images or oversized dimensions. Resize before sending.

Comparison table

Which to choose

Short prompts with one or two images, need low latency: Use OpenAI gpt-4o. The ecosystem and speed are unmatched.

Long documents with diagrams, context up to 200K, need tool use: Use Anthropic Claude 3 Sonnet. The returned token counts help you budget, and prompt caching reduces repeat costs.

Massive context (books, hours of video, thousands of images): Use Google Gemini 1.5 Pro. Nothing else fits 1M tokens without chunking.

Spanning providers without rewriting clients: Put an OpenAI-compatible gateway in front. Set a routing header to fall back when a provider is degraded. This avoids coupling your code to one wire format while keeping per-token metering.

Pick based on context length first, then cost, then ergonomics. The multimodal context window comparison providers above should make the trade-offs explicit before you commit.