OCR-heavy prompts: vision model latency benchmarks

Running an OCR vision model latency benchmark on document-heavy workloads reveals a counterintuitive truth: the slowest part is rarely the language model. Image encoding, transport, and prompt expansion dominate time-to-first-token, and model selection only matters once those are controlled.

The latency anatomy of OCR prompts

When you send a scan to a vision model, the request path looks like: client encodes image (base64 or URL) → gateway/provider decodes and runs vision encoder → encoder output projected to tokens → LLM prefill → generation. The prefill step scales with the number of vision tokens, which is a function of image resolution and the model’s patch size.

For a 1024×1024 PNG, GPT-4o-class models consume on the order of 500–1000 vision tokens; a 300 DPI A4 page cropped to 2048×2048 can hit 1500–2000. Those tokens must be processed by the transformer before the first text token emits. That prefill cost, not network round-trip, is what an OCR vision model latency benchmark should isolate.

Consider a 300 DPI scan of a standard US letter page: 2550×3300 pixels. No frontier model accepts that natively; they downsample or tile. Tiling strategies (used by some providers for high-res documents) multiply token count per tile. If a 512×512 tile yields ~256 vision tokens and the page splits into 20 tiles, you are at 5120 vision tokens before the LLM reads a single word. Prefill on a 70B-class transformer at that length is measurable in seconds, not milliseconds. This is why naive “send the PDF” scripts feel slow.

What an OCR vision model latency benchmark actually measures

Most published numbers mix cold-start, network, and inference. If you want actionable data, measure three things separately:

ttft (time to first token) from request send to first streamed token.
itl (inter-token latency) during text generation.
total_tokens returned, including vision token accounting.

A minimal Python client using the OpenAI-compatible interface:

import time, openai

client = openai.OpenAI(base_url="https://api.openai.com/v1", api_key="KEY")
img = "https://example.com/page.png"

t0 = time.perf_counter()
stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role":"user","content":[
        {"type":"image_url","image_url":{"url":img}},
        {"type":"text","text":"Transcribe all text verbatim."}
    ]}],
    stream=True,
)
first = None
for chunk in stream:
    if chunk.choices[0].delta.content:
        first = time.perf_counter()
        break
print("ttft", first - t0)

Swap base_url to a gateway and you can run the same script across models. n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models, so the same loop can benchmark Claude, Gemini, or open-weight VLMs without code changes.

Model classes and their tradeoffs

Multimodal LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro)

Pros: single call does OCR + reasoning, handles messy layouts, can answer follow-ups. Cons: high vision token cost, variable ttft under load. In our OCR vision model latency benchmark, these models showed the widest latency variance because provider queues prioritize text traffic.

Claude 3.5 Sonnet and Gemini 1.5 Pro both support million-token contexts, but that headroom does not reduce image prefill time; the vision encoder still runs per request. Gemini’s native multimodal input can batch multiple pages, yet each page adds linear tokens.

Specialized OCR services (Azure Document Intelligence, Google Vision, Tesseract)

Purpose-built pipelines skip the LLM prefill entirely. They return bounding boxes and text in hundreds of milliseconds for a page on commodity hardware (Tesseract local) or managed APIs. They cannot answer “summarize this invoice” without a second call. If your product only needs extracted text fields, this path wins on latency every time.

Open-source VLMs (LLaVA-1.6, Qwen-VL, InternVL)

Self-hosted gives you fixed latency if you control the GPU. Token counts are similar to closed models but you avoid network egress. Downside: accuracy on dense tables lags behind frontier models. You also own the scaling problem: a single A100 handles maybe a few images per second at 1024px.

Prompt and image preprocessing strategies

Resolution and cropping

Don’t send a 10 MB phone photo. Downscale to the model’s native resolution (often 768–1024 px short side) and crop whitespace. This cuts vision tokens quadratically.

from PIL import Image
im = Image.open("page.png").convert("RGB")
im.thumbnail((1024,1024))
im.save("page_small.jpg", quality=85)

Batching pages

If you have a 50-page PDF, parallelize requests rather than concatenating images into one giant message. A single 10-image multimodal message multiplies prefill latency and risks hitting context limits. Fire 10 parallel calls with a concurrency limiter.

Cache-control and reuse

Provider cache hints matter. If you send the same document image with different questions, set cache_control on the image content block so the vision encoder output is reused.

{
  "role": "user",
  "content": [
    {"type":"image_url","image_url":{"url":"https://.../page.png"},
     "cache_control": {"type":"ephemeral"}},
    {"type":"text","text":"What is the invoice total?"}
  ]
}

Gateways that forward provider cache-control hints pass this through, reducing repeat OCR vision model latency benchmark runs on the same asset.

Per-token metering exposes hidden costs

Latency and cost are coupled via token count. When a response is slow, check the usage object. n4n.ai provides per-token usage metering, which makes it straightforward to correlate a slow response with an unexpectedly high vision token count. If you see 4000 vision tokens for a seemingly small image, your encoder is tiling aggressively.

Common mistakes in benchmarking

Measuring only end-to-end wall clock. A 3s response might be 2.7s network retry.
Using a single 64px icon as test image. Real docs are high-res and dense.
Ignoring warm vs cold cache. First request to a provider often compiles graphs.
Letting auto-fallback swap models mid-test. Automatic fallback when a provider is rate-limited or degraded is great for prod, but it will silently change your model under test. Pin routing directives during benchmarks.

Benchmark methodology without fake numbers

To get reproducible data:

Use a fixed set of 20 document images (mix of printed, handwritten, tables).
Run each at three times of day to capture load variance.
Record ttft and total tokens; compute tokens/sec for generation separately.
Never average across models without noting vision token counts.

A simple table structure:

Model	Avg vision tokens	p50 ttft (s)	Notes
gpt-4o	~800	varies	queue sensitivity
local-llava	~700	stable	GPU-bound

(Values are illustrative placeholders; replace with your measured counts.)

Decision framework

Choose based on latency budget and task:

Sub-second hard limit, structured extraction: Use specialized OCR + cheap LLM for post-processing.
Complex layout + follow-up Q&A, can tolerate 2–5 s: Multimodal LLM direct.
Privacy/offline: Self-host VLM, downscale aggressively.

If you must use a multimodal endpoint, honor client routing directives to pin the model and avoid fallback surprises. The OCR vision model latency benchmark will only be comparable if the model is constant.

Takeaway

OCR latency is an image problem before it is a model problem. Measure vision token expansion, preprocess aggressively, and reserve multimodal LLMs for tasks where their reasoning pays the latency tax. For everything else, a dedicated OCR step followed by a text-only call will beat the all-in-one approach on both speed and cost.