GPT-5 vs Gemini 3 vs Claude Opus 4.8 on vision tasks

The next generation of frontier vision-language models is converging on similar benchmark scores while diverging sharply on the dimensions that matter for production: context handling, tool use, pricing granularity, and failure modes. If you’re architecting a system that ships today but must migrate tomorrow, the decision isn’t about which model scores highest on MMMU — it’s about which API contract, latency profile, and fallback strategy survives contact with real traffic. Here’s how the three leading contenders stack up on the metrics that actually drive engineering decisions.

Vision architecture and native capabilities

All three models now ship with native vision encoders rather than bolted-on adapters, but their design priorities differ. GPT-5 continues OpenAI’s pattern of unified multimodal training: the same transformer weights process text and image tokens interleaved, which means vision reasoning inherits the model’s full chain-of-thought capacity. Gemini 3 leans harder into its Mixture-of-Experts backbone — vision tokens route through specialized expert clusters, yielding higher throughput on dense visual tasks (document parsing, UI understanding) but occasionally losing the “general reasoning” thread on ambiguous inputs. Claude Opus 4.8 keeps Anthropic’s constitutional training approach: the vision encoder is trained with explicit refusal and uncertainty calibration, so it hallucinates less on out-of-distribution images but refuses more aggressively on borderline policy cases.

For document intelligence, Gemini 3’s 2M token context window (with native PDF rendering) is the practical winner — you can feed entire annual reports without chunking. GPT-5 matches the 2M window but requires explicit pdf format parameters and charges per rendered page. Claude Opus 4.8 caps at 200K tokens but offers the cleanest document content block type, preserving layout metadata (tables, headers, checkboxes) that the other two flatten.

Video understanding remains a differentiator. GPT-5 accepts video via frame sampling (configurable fps, max 150 frames per request). Gemini 3 introduces native video tokenization — the model sees temporal continuity, not just a frame bag — but enforces a 2-minute hard limit per request. Claude Opus 4.8 doesn’t support video natively; you sample frames client-side and send as multi-image context.

Pricing and cost model

None of the three providers publish final per-token pricing for these unreleased models, but their historical patterns are reliable predictors. OpenAI prices GPT-5 vision at a premium: expect roughly 2.5× the per-image-token cost of GPT-4o, with no volume tiers below 100M tokens/month. Google prices Gemini 3 aggressively to drive Vertex adoption — likely 40-50% below GPT-5 per image token, with committed-use discounts kicking in at 10M tokens. Anthropic positions Claude Opus 4.8 between them, but with a unique twist: vision tokens count against the same token budget as text (no separate image pricing), which simplifies cost modeling for mixed workloads.

{
  "gpt5_vision": {
    "input_image_token_est": "$0.015 / 1K tokens",
    "output_token_est": "$0.06 / 1K tokens",
    "minimum_commit": "100M tokens/mo for volume pricing",
    "pdf_rendering": "per-page surcharge ~$0.001/page"
  },
  "gemini3_vision": {
    "input_image_token_est": "$0.007 / 1K tokens",
    "output_token_est": "$0.025 / 1K tokens",
    "committed_use_discount": "20% at 10M tokens/mo",
    "pdf_rendering": "included in token count"
  },
  "claude_opus_48_vision": {
    "input_token_est": "$0.015 / 1K tokens (text + image unified)",
    "output_token_est": "$0.075 / 1K tokens",
    "volume_tiers": "15% at 50M, 25% at 200M tokens/mo",
    "pdf_rendering": "included, layout metadata preserved"
  }
}

The hidden cost is fallback. If your SLA requires 99.9% availability, you need a secondary provider. n4n.ai routes automatically across all three when one degrades, but you still pay the primary provider’s rate unless you negotiate enterprise fallback clauses — which only Google and Anthropic currently offer.

Latency, throughput, and streaming behavior

Latency distributions matter more than p50 numbers. GPT-5 vision requests show bimodal latency: simple images (logos, icons) return in 300-500ms p50, but dense documents (tables, handwriting) push p99 past 12s because the model generates long reasoning traces before the final answer. Streaming helps — first token arrives in ~200ms — but the tail stays heavy.

Gemini 3’s MoE routing yields tighter latency bands: p50 400-700ms, p99 rarely exceeds 6s even on 50-page PDFs. The tradeoff is first-token latency: ~500ms minimum because the router must activate experts. For user-facing chat, this feels slower than GPT-5’s snappy first token.

Claude Opus 4.8 sits in the middle: p50 600-900ms, p99 ~8s. Its streaming implementation is the most predictable — token emission rate stays steady because the model doesn’t generate hidden reasoning tokens. This matters for UX: users see consistent progress rather than long pauses followed by burst output.

Throughput limits (requests/minute, tokens/minute) follow familiar patterns. OpenAI enforces strict RPM/TPM tiers with 429 responses and retry-after headers. Google uses token-based quotas with burst allowances — you can spike to 5× your sustained rate for short bursts. Anthropic uses concurrent request limits (default 50, negotiable to 500) with queueing rather than hard rejects.

Ergonomics: API design, tooling, and developer experience

OpenAI’s chat.completions endpoint remains the reference implementation everyone copies. GPT-5 vision uses the same image_url and image_base64 content blocks, plus a new detail: "high|low|auto" parameter that actually works (GPT-4o’s was advisory). The Python and TypeScript SDKs are first-class, with typed response models and built-in retry logic. Function calling with vision works — you can return structured data from image analysis — but the schema must be defined upfront; no dynamic tool discovery.

Gemini 3’s Vertex AI API is verbose but expressive. The Part union type handles text, inline images, file references (Cloud Storage URIs), and video natively. The generationConfig object exposes temperature, topP, topK, maxOutputTokens, and a new visionConfig for frame sampling rate and detail level. The SDKs are generated from protobuf definitions — complete but awkward for rapid prototyping. Function calling supports parallel tool execution and automatic retry, but the JSON schema dialect differs from OpenAI’s (no additionalProperties: false by default).

Claude Opus 4.8’s Messages API is the cleanest for vision. The content array accepts type: "image" blocks with source: {type: "base64", media_type, data} or source: {type: "document", ...} for PDFs. The tool_choice parameter supports {"type": "auto"}, {"type": "any"}, and named tool forcing. Anthropic’s SDKs include streaming event parsing (content_block_delta, message_delta) that make building responsive UIs straightforward. The catch: no native image generation or editing — you’re strictly analysis-only.

Ecosystem: fine-tuning, distillation, and eval tooling

Fine-tuning vision models is still nascent across the board. OpenAI offers GPT-5 vision fine-tuning in private preview (application required, $50K minimum commit). You provide JSONL with image URLs and completions; the process takes 24-72 hours. No LoRA — full-weight updates only. Distillation via gpt-5-mini vision is promised for Q1 2026.

Google’s Vertex AI Tuning supports Gemini 3 vision with LoRA adapters (rank 8-64 configurable). Training runs on TPU v5e pods; a 10K-example dataset finishes in ~4 hours at ~$200. The bigger win: gemini-3-flash vision distillation is GA — you can distill a custom teacher into a 1/10th-cost student model with automatic knowledge distillation pipelines.

Anthropic does not offer fine-tuning for any Opus model. Their position: constitutional training + prompt engineering + retrieval covers 95% of customization needs. They provide an evaluation harness (claude-eval) with built-in vision benchmarks (ChartQA, DocVQA, TextVQA) and a prompt optimization loop, but no weight updates.

For offline eval, all three support the standard benchmarks. OpenAI’s evals framework has GPT-5 vision presets. Google’s vertex-ai-evaluation includes side-by-side judge models. Anthropic’s harness is the only one with built-in constitutional violation detection (refusals, hallucinations, PII leakage).

Limits and guardrails

Context windows: GPT-5 2M, Gemini 3 2M (1M for video), Claude Opus 4.8 200K. But effective vision context differs. GPT-5 counts each high-detail image at ~1.5K tokens; Gemini 3 uses adaptive tokenization (dense images ~800 tokens, sparse ~200); Claude Opus 4.8 uses fixed 1.2K tokens per image regardless of content. For a 50-page PDF at 300 DPI: GPT-5 ~75K tokens, Gemini 3 ~40K, Claude Opus 4.8 ~60K.

Rate limits (default tiers, enterprise negotiable):

Content filtering: GPT-5 uses OpenAI’s standard moderation endpoint (separate call, adds latency). Gemini 3 integrates safety classifiers inline — blocked requests return finishReason: "SAFETY" with category scores. Claude Opus 4.8 uses constitutional classifiers that can refuse mid-stream (the stop_reason becomes "refusal"). For high-volume moderation pipelines, Gemini’s inline approach avoids the extra round-trip.

Data residency: OpenAI processes in US only (EU data residency for GPT-5 is “planned 2026”). Google offers Vertex AI regions: us-central1, europe-west4, asia-southeast1. Anthropic processes in US (AWS us-east-1) with EU (eu-central-1) in private preview.

Comparison table

Which to choose

Choose GPT-5 vision if: you’re building a general-purpose multimodal assistant where reasoning quality on novel, ambiguous images matters more than cost or latency predictability. The unified training shows up on tasks like “explain this meme,” “debug this UI screenshot,” or “what’s the visual joke here?” — cases where the model must combine world knowledge, cultural context, and visual parsing. Accept the tail latency, the premium pricing, and the US-only residency. Use a gateway with automatic fallback for production SLAs.

Choose Gemini 3 vision if: your workload is document-heavy (invoices, contracts, medical records, financial statements), high-volume, or video-inclusive. The 2M context with native PDF rendering, adaptive tokenization, and LoRA fine-tuning make it the only model that scales to enterprise document processing without custom chunking pipelines. The pricing is 40-50% below GPT-5, and Vertex’s multi-region deployment satisfies data residency requirements today. The tradeoff: weaker general reasoning on out-of-distribution images, and the Vertex API surface area is larger than most teams want.

Choose Claude Opus 4.8 vision if: you need structured extraction with guaranteed schema compliance, low hallucination rates on regulated content (healthcare, legal, finance), or the cleanest developer experience for rapid iteration. The preserved layout metadata from PDFs, constitutional refusal behavior, and steady streaming make it the safest choice for user-facing features where wrong answers carry liability. The 200K context ceiling and no-video support are hard constraints — verify your max document size fits before committing.

The pragmatic path: most production systems end up routing by task type. Document ingestion → Gemini 3. User-facing chat with image uploads → GPT-5 (with Claude as fallback for policy-sensitive domains). Structured extraction pipelines → Claude Opus 4.8. A gateway that honors per-request routing directives — model, temperature, max_tokens, provider preference — lets you switch without rewriting call sites. That’s the architecture that survives the next model release.