Multimodal API rate limits: images vs text tokens

Most teams budget for text tokens and treat image inputs as an afterthought, but multimodal API rate limits images vs text are calculated on entirely different axes. The result is silent throttling, blown quotas, and unpredictable latency when a vision-heavy request hits a provider’s hidden token bucket.

The asymmetry in multimodal API rate limits images vs text

Text tokens are a direct function of your prompt length. Image tokens are a function of pixel geometry, provider-specific resampling, and a conversion constant that is rarely the same across vendors. Both count against the same transactions-per-minute (TPM) or requests-per-minute (RPM) ceiling, but the weight of an image is invisible until you read the fine print.

This asymmetry breaks naive rate-limit logic. A service that sends 20 high-resolution photos with a one-line caption can blow a 30k TPM limit while consuming fewer than 200 text tokens. The rate limiter does not care that the tokens came from pixels.

Engineers comparing providers on price-per-text-token miss the dominant cost driver in vision workloads. You must model multimodal API rate limits images vs text as a composite token stream with per-provider image multipliers.

How providers convert images to tokens

OpenAI’s tile model

OpenAI (GPT-4V, GPT-4o) uses a fixed base plus a tile grid. For low detail, an image is a flat 85 tokens. For high detail, the image is split into 512×512 tiles; each tile costs 170 tokens, plus the 85 base.

def openai_image_tokens(width, height, detail="high"):
    if detail == "low":
        return 85
    tiles_x = (width + 511) // 512
    tiles_y = (height + 511) // 512
    return 85 + 170 * tiles_x * tiles_y

A 1024×1024 image at high detail is 4 tiles → 85 + 680 = 765 tokens. A 2048×1536 image is 4×3 = 12 tiles → 85 + 2040 = 2125 tokens.

Anthropic’s area model

Anthropic resizes the image so its longest side fits within 1568px, then maps the resulting pixel area to tokens at a fixed ratio documented in their cookbook. The key difference: no discrete tile boundaries, so cost scales more smoothly with resolution. A medium 800×600 photo after resize stays roughly that size and consumes on the order of a few hundred to low-thousands of tokens depending on model family.

The practical consequence: for small images, Anthropic is often cheaper in token count than OpenAI’s tile model; for very large images, the area model can surpass tile cost because OpenAI caps tile count via downsampling in high-detail mode only if you explicitly request it.

Gemini’s approach

Gemini folds images into the multimodal token stream using a resolution-dependent block estimate. The exact constant is version-specific, but the pattern holds: token cost grows with pixel count, not with a fixed per-image fee. When you compare multimodal API rate limits images vs text across these three, the only constant is that images are never “one token.”

Rate limit buckets: where images hide

Providers expose limits in different shapes:

• Unified TPM: OpenAI counts image-derived tokens inside the same TPM as text. If your org limit is 30k TPM, a single 25-image batch can exhaust it.
• Separate vision quotas: Some enterprise tiers expose a distinct vision TPM or image RPM. Ignore these at your peril; the gateway may reject a request that looks fine on text TPM.
• Request-level caps: Anthropic may rate-limit by total input size per request, which implicitly caps images-per-call even when TPM is available.

The trap is that your client library reports remaining_tokens based on text only. You need to subtract image estimates before issuing the call.

Concrete example: sending 10 high-res photos

Assume a warehouse inspection app sends 10 photos at 2048×1536, high detail, with a 50-character caption.

OpenAI token math:

• Per image: 2125 tokens
• 10 images: 21,250 tokens
• Text: ~12 tokens
• Total: 21,262 tokens

If your TPM limit is 30,000, you can send at most ~14 such images per minute. Two batches per minute triggers a 429. The caption is irrelevant.

Anthropic equivalent (area model, resized to 1568×1176 = 1.84M pixels, ~1 token per 750 pixels): ~2450 tokens/image → 24,500 tokens. Similar envelope, different constant.

This is why multimodal API rate limits images vs text must be planned as image-dominated budgets.

Code: estimating token load before send

Build the estimator into your request path. Below is a minimal client-side guard:

def estimate_openai_request(text, images):
    text_tok = len(text) // 4
    img_tok = sum(openai_image_tokens(w, h, d) for (w, h, d) in images)
    return text_tok + img_tok

def guard_call(text, images, tpm_budget, used_tpm):
    needed = estimate_openai_request(text, images)
    if needed + used_tpm > tpm_budget:
        raise ValueError(f"Image-heavy call needs {needed} tokens, only {tpm_budget - used_tpm} left")

When you pass the request to an inference gateway, include routing hints so fallback targets a provider with a different image formula:

{
  "model": "gpt-4o",
  "messages": [
    {"role": "user", "content": [
      {"type": "image_url", "image_url": {"url": "https://cdn/photo.jpg", "detail": "high"}},
      {"type": "text", "text": "List defects"}
    ]}
  ],
  "route": {"prefer": ["openai"], "fallback": ["anthropic"]},
  "cache_control": {"type": "ephemeral"}
}

When you send this through n4n.ai, it honors the route directive and forwards cache-control, but the per-token metering will still reflect the image math above—you cannot escape the token count by routing.

Tradeoffs of normalization

Treating images as token equivalents simplifies accounting but hides real tradeoffs:

• Detail level: Dropping from high to low detail on OpenAI cuts 2125 tokens to 85. That is a 25× reduction for possibly acceptable quality on small objects.
• Latency: Image preprocessing (resize, tile, encode) adds seconds before the model sees tokens. A cheaper token count may cost more wall-clock time.
• Accuracy: Underspending on image tokens loses visual detail. For OCR, low detail fails; for scene classification, it may suffice.

You should parameterize detail per use case, not globally.

Routing and fallback strategies

Automatic fallback is essential because provider degradation is common during peak vision load. But fallback changes the token equation:

1. Primary OpenAI rejects at 21k tokens used.
2. Fallback Anthropic accepts, but its area model bills ~24.5k tokens for the same images.
3. Your cost dashboard shows a spike not because the image changed, but because the token constant changed.

Set fallback only when your token budget accounts for the worst-case target provider. Gate fallback with the estimator above, and tag requests with expected token class so metering can alert on drift.

A gateway that provides automatic fallback when a provider is rate-limited or degraded reduces 429s, but it does not absolve you of modeling multimodal API rate limits images vs text. The fallback path must be provisioned with its own token ceiling.

Takeaway

Image inputs are the dominant token consumer in virtually every vision-enabled LLM call. Model them explicitly with provider-specific formulas, subtract them from your TPM budget before send, and tune detail level per task. Routing and fallback mitigate throttling but introduce cross-provider token variance—provision for the highest-cost path. Teams that treat multimodal API rate limits images vs text as a unified token stream will ship stable vision features; those that treat images as free requests will chase 429s in production.