Multi-image prompts: latency benchmarks across vision models

Multi-image prompt latency vision models is rarely dissected with the same rigor as text completion benchmarks, yet it dictates real user experience in any multimodal product. When you attach three screenshots to a single LLM call, the time-to-first-token behaves nothing like the sum of independent single-image requests, and the variance across providers is large enough to change your architecture.

Why image packing is not free

A vision LLM does not “see” an image the way a human flips through a deck. Each image is preprocessed by a vision encoder, projected into token embeddings, and concatenated into the sequence that the language model attends to. The fixed cost of that encode step is paid per image, and the marginal cost of attention grows with total sequence length.

OpenAI documents that GPT-4o breaks images into 512px tiles, each costing 85 tokens plus a per-image overhead. A 1024×1024 screenshot becomes four tiles; three such images inject roughly 12 tiles and over a thousand tokens before you write a single word of prompt text. That token count directly drives prefill latency.

How architectures handle multiple images

Proprietary API models

GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro all accept a list of image URLs or base64 blobs in a single message. Internally they differ:

GPT-4o runs a shared vision encoder per image, then concatenates. Prefill cost scales with tile count.
Claude resizes to a standard grid and embeds each image as a fixed set of tokens; latency tracks image count closely.
Gemini 1.5 treats images as native multimodal tokens in a long context. With prefix caching enabled, repeated images in a conversation can be cheaper, but the first multi-image prefill still pays per pixel.

Open-weight vision models

Self-hosted stacks (Llava, Qwen-VL, Molmo) typically use a ViT encoder followed by a projection layer. The encoder forward pass is often the bottleneck on GPU-bound servers. Unlike hosted APIs, you feel the queueing delay directly: two images mean two encoder batches unless you explicitly pack them.

# Typical local call with Hugging Face transformers
from transformers import LlavaForConditionalGeneration, AutoProcessor
import torch

model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf").to("cuda")
processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf")

# Two images -> two forward passes through vision tower if not batched
inputs = processor(images=[img1, img2], text=prompt, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=200)

If your serving layer batches the vision encoder, multi-image latency can stay near single-image; if not, it doubles.

The non-linearity you should expect

Latency for multi-image prompt latency vision models is not a simple linear function of image count. Two effects dominate:

Encoder fixed overhead – each image triggers a full forward pass through the vision tower. This is pure additive cost.
Attention prefill – language models compute attention over the full token sequence during prefill. Image tokens are numerous; going from 1 to 4 images can expand the sequence by 3–5×, and prefill time grows faster than linearly on transformers without sparse attention.

A small qualitative observation from repeated runs: models with separate vision encoders show steep latency curves past three images, while architectures that share a fused multimodal backbone (Gemini) degrade more gracefully. None of them are “free” past the first image.

Measure it yourself

Do not trust vendor marketing. Write a loop that varies image count and records time-to-first-token (TTFT) and total latency.

import time, openai, asyncio

client = openai.OpenAI(base_url="https://api.openai.com/v1", api_key="KEY")

async def timed_call(n_images):
    urls = [{"type": "image_url", "image_url": {"url": f"https://picsum.photos/1024/{i}"}}
            for i in range(n_images)]
    content = [{"type": "text", "text": "Describe differences."}] + urls
    start = time.perf_counter()
    resp = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": content}],
        stream=True,
    )
    first = None
    for chunk in resp:
        if chunk.choices[0].delta.content:
            first = time.perf_counter()
            break
    total = time.perf_counter() - start
    return first - start, total

for n in [1, 2, 4, 8]:
    ttft, tot = timed_call(n)
    print(f"{n} imgs: ttft={ttft:.2f}s total={tot:.2f}s")

Run this against each candidate model. The shape of the curve is your real benchmark.

Batching vs parallel single-image calls

Engineers default to multi-image prompts because the API makes it easy. Sometimes that is wrong.

Use a single multi-image call when:

The task requires cross-image reasoning (“Which diagram has the newer schema?”).
You need the model to attend to both images simultaneously to avoid contradictions.

Use parallel single-image calls when:

Images are independent (OCR each receipt, then aggregate in code).
You care about tail latency; one stuck image encode shouldn’t block the others.
Your serving stack penalizes long sequences.

The tradeoff is reasoning quality versus predictable latency. A separate-call pipeline lets you cap each request at 2 seconds and fan out, but you lose the model’s global attention.

Routing and degradation

In production, provider outages or rate limits turn latency from a constant into a cliff. An OpenAI-compatible gateway such as n4n.ai that honors client routing directives can shift a multi-image workload to a fallback model when the primary is degraded, but the underlying latency profile remains model-specific—fallback is not a latency fix, just availability insurance.

Per-token metering matters here: a fallback model that is slower but cheaper may be acceptable if the image token blowup is visible in your billing data and you can throttle.

A decision guide

1–2 images, cross-image reasoning: single call to a fused-backbone model (Gemini 1.5) or GPT-4o.
3–5 images, independent tasks: parallel single-image calls, aggregate in your app.
>5 images: avoid one giant prompt. Use retrieval or downscale; consider a model with native long context and test prefill cost first.
Self-hosted: ensure your vision encoder batches across images, or latency will scale linearly with count.

Takeaway

Multi-image prompt latency vision models is a function of encoder overhead and sequence length, not image pixels alone. Batching images saves engineering effort but hides a latency tax that grows steeply on encoder-separated architectures. Measure the curve for your actual image sizes, prefer separate calls for independent work, and reserve true multi-image prompts for tasks that need joint attention. Pick the model based on its latency shape, not its leaderboard score.