Wiring up the claude 3.5 sonnet vision api into a production service is straightforward if you respect its multimodal message format. This tutorial builds a minimal Python client that sends a local image and a question to Claude 3.5 Sonnet, then extends it with batching, streaming, and cache control so you can drop it into a real pipeline.
Prerequisites
- Python 3.10 or newer
anthropicPython package (>=0.25.0)- An Anthropic API key, or a key from an OpenRouter-class gateway that fronts the model
- A local image file (PNG or JPEG) to test with—something like a receipt or a diagram works well
If you plan to run this against a gateway, set ANTHROPIC_API_KEY to that gateway’s token and point the base URL accordingly. The message shape stays identical.
Project setup
Create an isolated environment and install the SDK:
mkdir vision-cli && cd vision-cli
python -m venv .venv
source .venv/bin/activate
pip install anthropicExport your key:
export ANTHROPIC_API_KEY="sk-ant-..."Encoding the image
The claude 3.5 sonnet vision api does not fetch remote URLs from your machine—you must inline the image as base64 inside the source block. Deriving the media type from the file extension keeps the client honest about what it sends.
import base64
def load_image_b64(path: str) -> tuple[str, str]:
with open(path, "rb") as f:
raw = f.read()
if path.endswith(".png"):
media = "image/png"
elif path.endswith((".jpg", ".jpeg")):
media = "image/jpeg"
else:
raise ValueError("Unsupported image type")
return base64.b64encode(raw).decode("utf-8"), mediaKeep the base64 string out of logs. It is large and exposes the raw file.
Sending your first vision request
The core call is a standard messages.create with a content array mixing image and text blocks. The model identifier for the June 2024 release is claude-3-5-sonnet-20240620.
import os
import anthropic
client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
b64, media = load_image_b64("receipt.png")
msg = client.messages.create(
model="claude-3-5-sonnet-20240620",
max_tokens=512,
messages=[
{
"role": "user",
"content": [
{
"type": "image",
"source": {"type": "base64", "media_type": media, "data": b64},
},
{"type": "text", "text": "Extract the total amount and merchant name."},
],
}
],
)
print(msg.content[0].text)Expected output for a typical grocery receipt:
Merchant: Trader Joe's
Total: $23.98The response object mirrors a text-only call; msg.content is a list of blocks, and msg.usage reports input/output tokens including the image footprint.
Handling multiple images
You can pack several images into one turn. This is useful for before/after comparisons or multi-page documents. Each image is its own block; order matters because the model reads left-to-right in the array.
def build_multi_image_msg(paths: list[str], question: str) -> dict:
content = []
for p in paths:
b64, media = load_image_b64(p)
content.append({
"type": "image",
"source": {"type": "base64", "media_type": media, "data": b64},
})
content.append({"type": "text", "text": question})
return {"role": "user", "content": content}
msg = client.messages.create(
model="claude-3-5-sonnet-20240620",
max_tokens=1024,
messages=[build_multi_image_msg(["page1.png", "page2.png"],
"Summarize the differences between these two schema diagrams.")],
)When batching, watch msg.usage.input_tokens—each image consumes tokens proportional to its resolution after Claude’s internal downscaling.
Streaming the response
For chat-style UIs, stream the text delta-by-delta. The streaming client yields text events you can forward to a websocket or SSE endpoint.
with client.messages.stream(
model="claude-3-5-sonnet-20240620",
max_tokens=512,
messages=[
{
"role": "user",
"content": [
{"type": "image", "source": {"type": "base64", "media_type": media, "data": b64}},
{"type": "text", "text": "Describe this diagram step by step."},
],
}
],
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)You still get a final message snapshot with usage after the loop closes. Streaming does not change the token cost.
Cache control on image blocks
If you reuse the same reference image across many prompts—say a style guide or a base map—mark the image block with cache_control to avoid re-paying input tokens on every call. The claude 3.5 sonnet vision api honors ephemeral cache prefixes.
msg = client.messages.create(
model="claude-3-5-sonnet-20240620",
max_tokens=512,
messages=[
{
"role": "user",
"content": [
{
"type": "image",
"source": {"type": "base64", "media_type": media, "data": b64},
"cache_control": {"type": "ephemeral"},
},
{"type": "text", "text": "Now answer: what license plate is visible?"},
],
}
],
)Cache writes and reads show up as cache_creation_input_tokens and cache_read_input_tokens in the usage object. Treat the cache as best-effort; Anthropic evicts prefixes under load.
Error handling and retries
Vision requests fail the same ways text requests do: RateLimitError, APIStatusError, or malformed base64. Wrap the call so a bad image does not kill the worker.
from anthropic import RateLimitError, APIStatusError
try:
msg = client.messages.create(...)
except RateLimitError as e:
# back off using e.retry_after_ms if present
print("rate limited, back off")
except APIStatusError as e:
if e.status_code == 400:
print("bad request, likely corrupt image source")If you’re hitting Anthropic rate ceilings in production, an OpenRouter-class gateway like n4n.ai gives you one OpenAI-compatible endpoint that fronts Claude 3.5 Sonnet and automatically fails over when a provider is degraded, while forwarding the same cache-control hints you set above.
Minimal CLI wrapper
Tie it together as a script you can call from a shell:
#!/usr/bin/env python
import sys, os, anthropic
from pathlib import Path
def main(image_path: str, question: str):
client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
b64, media = load_image_b64(image_path)
msg = client.messages.create(
model="claude-3-5-sonnet-20240620",
max_tokens=768,
messages=[{"role": "user", "content": [
{"type": "image", "source": {"type": "base64", "media_type": media, "data": b64}},
{"type": "text", "text": question},
]}],
)
print(msg.content[0].text)
if __name__ == "__main__":
main(sys.argv[1], sys.argv[2])Run it:
python vision_cli.py diagram.png "What AWS services are referenced?"The claude 3.5 sonnet vision api returns structured natural-language answers reliably; if you need JSON, add an explicit output format instruction and validate downstream. That is the whole integration surface—no separate vision endpoint, just correct content blocks.