Sending images to GPT-4o and Claude: an API comparison

To send images gpt-4o claude api, you hit two similar but subtly different request shapes. Both accept multimodal inputs and let you interleave text and visuals, but the wire format, token accounting, and failure modes differ in ways that matter once you build a production pipeline.

Capabilities

What each model accepts

GPT-4o exposes vision through the Chat Completions endpoint. You pass images as image_url content blocks inside a multipart content array. It handles multiple images per turn and can reason across them.

Claude (using the Messages API, e.g. claude-3-5-sonnet) treats images as discrete image content blocks. Each block carries a source that is either a url or base64 data with an explicit media_type. Claude also supports PDFs as documents, but for pure vision the image block is the primitive.

Both models accept PNG and JPEG. Claude additionally accepts WebP and GIF; OpenAI’s documented set covers PNG, JPEG, and WebP. Neither lets you embed an image mid-token-stream; you stage it in the message structure.

Reasoning behavior

GPT-4o is tuned for fast multimodal turns and handles dense charts or UI screenshots well. Claude tends to produce longer structured descriptions and is strong at following explicit extraction schemas. Neither is a substitute for OCR post-processing if you need exact text recall.

Wire format and ergonomics

The fastest way to see the difference is code.

from openai import OpenAI

client = OpenAI()
resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Extract the total from this receipt"},
            {"type": "image_url", "image_url": {"url": "https://example.com/receipt.jpg"}}
        ]
    }]
)
print(resp.choices[0].message.content)

import anthropic

client = anthropic.Anthropic()
resp = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Extract the total from this receipt"},
            {"type": "image", "source": {
                "type": "url",
                "url": "https://example.com/receipt.jpg"
            }}
        ]
    }]
)
print(resp.content[0].text)

If you need base64 (common for private data), Claude requires media_type:

{
  "type": "image",
  "source": {
    "type": "base64",
    "media_type": "image/png",
    "data": "iVBORw0KGgo..."
  }
}

OpenAI uses a data URI inside image_url.url:

{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}}

When you send images gpt-4o claude api through a unified client, the translation layer is mostly mechanical but watch the max_tokens requirement: Claude rejects requests without it; OpenAI has a default.

Cost model

At launch, OpenAI priced GPT-4o text at $5 per 1M input tokens and $15 per 1M output. Claude 3.5 Sonnet listed at $3 per 1M input and $15 per 1M output. Images are not free: both count image pixels as input tokens.

OpenAI documents image token cost as a function of resolution (tiles of 512px). Claude counts image input tokens similarly based on dimensions. A 1024×1024 PNG is not a fixed price across both—you must measure with their token estimators or your own proxy logs.

If you route through an OpenAI-compatible gateway such as n4n.ai, you still pay the underlying provider rate, but per-token usage metering lets you attribute image-heavy calls per tenant without building your own accounting.

Latency and throughput

Both providers stream. GPT-4o typically opens the stream quickly on small images; Claude’s first-token latency is comparable for single-image prompts. Throughput degrades with image count and raw pixel area, not just output length.

Practical observation: resizing a 4000px wide photo to 1024px before sending cuts latency more than dropping max_tokens from 1024 to 512. Both APIs honor stream: true, but Claude requires you to buffer message_start and content_block_delta events; OpenAI gives you chunk.choices[0].delta.

Ecosystem and tooling

OpenAI’s SDK is ubiquitous. Every LLM orchestration library speaks Chat Completions. If your stack already uses LangChain, Haystack, or a homegrown client, GPT-4o drops in.

Anthropic’s first-party SDK is clean and typed, but you’ll write more adapter code to fit it into OpenAI-centric pipelines. The Messages API does not support system messages inside messages; you pass a top-level system field. That trips up naive proxies.

For fallback, a gateway that honors client routing directives and forwards provider cache-control hints saves you from 429 storms. You can send the same image payload to either backend if you normalize the content blocks.

Limits

• Max images per request: OpenAI allows multiple image_url blocks; Claude allows multiple image blocks. Both practically cap on total token context (128K for GPT-4o, 200K for Claude 3.5 Sonnet).
• File size: OpenAI rejects oversized data URIs; Claude limits base64 payloads per request. Use hosted URLs when possible.
• Formats: Stick to PNG/JPEG unless you’ve tested WebP on your target model.
• Rate limits: Image requests consume the same RPM as text but weigh heavier on TPM.

Comparison table

Which to choose

Existing OpenAI stack, low tolerance for adapter code Use GPT-4o. The send images gpt-4o claude api decision is trivial when your retriever already emits image_url blocks. You avoid a second SDK and keep one token accounting path.

Long-context document vision with large scans Claude’s 200K window and explicit system field suit batch extraction from many page images. You can pack more slides before hitting context limits.

Cost-sensitive high-volume classification Claude’s lower input token rate helps if your prompts are image-heavy and output-short. Benchmark with your own resolution presets; don’t trust list price alone.

Strict data handling with private base64 Both accept base64, but Claude’s mandatory media_type forces disciplined encoding. If you already sanitize mime types upstream, either works; if not, GPT-4o’s data URI is more forgiving.

Need automatic fallback If a provider is degraded, a routing layer that speaks OpenAI format to your code and translates to Claude on failure keeps your pipeline green. That’s the only scenario where the abstraction pays for itself on day one.

Pick the model whose request shape matches your existing message builder. The vision quality gap is small; the integration tax is where the real cost lives.