Vision API rate limits and image size constraints compared

When you wire a multimodal model into production, the two constraints that bite first are vision api rate limits image size. OpenAI, Anthropic, and Google each expose vision capabilities through similar chat completions interfaces, but the way they meter requests and resize uploaded pixels diverges in ways that affect throughput, cost, and error handling.

The contenders

We compare three hosted APIs that accept raw images alongside text:

OpenAI gpt-4o (and legacy gpt-4-vision-preview)
Anthropic claude-3-opus / claude-3-sonnet with vision
Google gemini-1.5-pro multimodal

All three accept base64 or URL image inputs in a chat turn. None accept video streams in the same call (Google does in separate API). The differences show up when you push volume or resolution.

How image size becomes tokens

Every provider converts pixels to tokens, but the math differs.

OpenAI uses server-side downsampling. The longest side is scaled to 2048px (or 768px for low-detail mode), then split into 512px tiles. Each tile costs 170 tokens, plus an 85-token base. This caps worst-case token blowup: a 6000x4000 RAW becomes 2048x1365, yielding 4x3=12 tiles → 2125 tokens.

def openai_image_tokens(width, height, low_detail=False):
    if low_detail:
        return 85
    scale = min(2048 / max(width, height), 1)
    w, h = int(width*scale), int(height*scale)
    tiles_x = (w + 511) // 512
    tiles_y = (h + 511) // 512
    return tiles_x * tiles_y * 170 + 85

Anthropic counts patches without forced resize. A 16x16 pixel block is roughly one token. Images above 20MB are rejected. A 4000x3000 image is ~7500 tokens before any text. That scales linearly with area, so 4K scans get pricey fast.

def anthropic_image_tokens(width, height):
    # approximate, ignores overhead
    return (width // 16) * (height // 16)

Google Gemini normalizes uploads to a max resolution (longest side capped well above 2K) and uses a patch-based encoder. Exact token formula is undocumented, but area dominates. In practice a 2048px image costs fewer tokens than on Anthropic but more than OpenAI low-detail.

The practical upshot: a 4000x3000 photo costs ~1700 tokens on OpenAI (high detail), ~7500 on Anthropic, and an unknown but comparable amount on Gemini. Those tokens count against context windows and rate limit quotas.

Vision api rate limits image size policies

Rate limits are enforced on requests per minute (RPM) and tokens per minute (TPM). Image tokens inflate TPM fast.

OpenAI ties limits to usage tiers. A new account may see 10–20 RPM on gpt-4o; image requests consume TPM like any other. Exceeding returns HTTP 429 with retry-after. The platform also enforces max image dimensions implicitly via the 20MB cap.

Anthropic sets per-organization RPM/TPM that scale with trust tier. A single 20MB image can eat a large slice of a 100k TPM limit. Their gateway rejects URLs; you must base64, which adds client CPU and memory pressure.

Google uses project-level QPM (queries per minute) and TPM. Gemini’s 1M-token context means a single image+text call rarely hits TPM, but QPM still throttles bursty workloads. Google also rate-limits by concurrent requests differently per region.

If you front these with a unified endpoint such as n4n.ai, you get automatic fallback when a provider is rate-limited or degraded, but you still must respect the underlying vision api rate limits image size of the routed model. The gateway forwards provider cache-control hints, so repeated identical images can hit provider-side caches if you set them.

Throughput and latency reality

Image preprocessing dominates client latency. OpenAI’s URL fetch moves bytes server-side; Anthropic and Google require you to ship base64, adding ~33% size overhead on the wire. For a 5MB JPEG, that’s 6.6MB upload before any inference.

Inference latency correlates with token count. OpenAI’s tiling keeps token count bounded, so p95 latency stays flat across resolutions. Anthropic latency grows with pixel area. Gemini sits in between but benefits from TPU batching.

A sane client pattern:

import time, random
def call_with_backoff(fn, max_retries=5):
    for i in range(max_retries):
        try:
            return fn()
        except RateLimitError as e:
            wait = (2**i) + random.random()
            time.sleep(wait)
    raise

Cost model

All three price per input and output token. Image tokens are input tokens.

OpenAI gpt-4o: ~$2.50 / 1M input, ~$10 / 1M output (known public pricing).
Anthropic claude-3-sonnet: ~$3 / 1M input, ~$15 / 1M output.
Google gemini-1.5-pro: ~$1.25 / 1M input for first 128k, higher beyond.

Because Anthropic charges more per image pixel, high-res document scans get expensive. OpenAI’s tiling caps worst-case cost. Gemini is competitive for mixed long-context. Example: 10k images/month at 2000px avg. OpenAI ~$0.05/ image input; Anthropic ~$0.15; Gemini ~$0.03.

Ergonomics and SDKs

OpenAI’s SDK is simplest for image URLs:

{
  "model": "gpt-4o",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "image_url", "image_url": {"url": "https://x/y.jpg"}},
      {"type": "text", "text": "Describe"}
    ]
  }]
}

Anthropic requires base64 with media type:

import base64
b64 = base64.b64encode(open("img.jpg","rb").read()).decode()
# pass as {"type":"image","source":{"type":"base64","media_type":"image/jpeg","data":b64}}

Google uses inline_data:

const req = {
  contents: [{
    parts: [
      { inlineData: { mimeType: "image/jpeg", data: b64 } },
      { text: "Describe" }
    ]
  }]
};

All three support function calling alongside images, but OpenAI and Google have mature tool ecosystems; Anthropic’s is newer. OpenAI also offers detail: low flag to halve cost.

Ecosystem and limits beyond size

OpenAI: largest third-party tooling, prompt caches, and fine-tunes. Anthropic: strong long-document vision, strict content moderation. Google: native multimodal with audio/video separate, deep GCP integration.

Hard limits to remember:

OpenAI: 20MB per image, URL must be fetchable.
Anthropic: 20MB per image, no URL fetch—must base64.
Google: 20MB per file, supports PDF with images.

None permit client-side token compression; you must resize before send if you want to save cost.

Comparison table

Dimension	OpenAI `gpt-4o`	Anthropic `claude-3`	Google `gemini-1.5-pro`
Capabilities	Text+image chat, tools	Text+image chat, tools	Text+image (separate audio/video)
Max image size	20MB, longest side 2048px (auto)	20MB, no resize	20MB, normalized to ~3k px
Image token formula	Tiles 512px → 170/tile +85	~1 token / 16x16 patch	Patch-based, undocumented
Rate limit basis	RPM + TPM per tier	RPM + TPM per org	QPM + TPM per project
Cost input / 1M	~$2.50	~$3 (sonnet)	~$1.25 (first 128k)
SDK ergonomics	URL or b64, simplest	b64 only	b64 only, typed SDK
Fallback options	Single provider	Single provider	Single provider

Which to choose

Low-latency thumbnail tagging: OpenAI gpt-4o with low-detail mode. Small images, predictable token cost, mature SDK.

High-resolution document understanding: Anthropic Claude if you need fine patch detail and can afford token cost; Google Gemini if you want long context alongside the image.

Cost-sensitive bulk classification: Google Gemini on first-tier pricing, or OpenAI with aggressive client-side downscaling to 768px.

Multi-provider resilience: Route through a gateway that honors client routing directives and forwards provider cache-control hints. A gateway like n4n.ai simplifies multi-provider routing but doesn’t relax the fundamental vision api rate limits image size. Implement exponential backoff on 429 and cache image tokens client-side.

Pick based on where your pixels land: downsized and frequent → OpenAI; detailed and sparse → Anthropic; massive context and mixed → Gemini.