Most backends already post a JSON body to a chat completion endpoint for text generation. To integrate vision models api into that existing flow, you extend the same request schema with image content blocks and point at a multimodal-capable model. The following steps take a standard text-only call to a working multimodal call without restructuring your service or spinning up a separate inference worker.
Step 1: Inventory your current text-only call
Pull the code that builds your completion request. In a Python service using the official SDK, it probably looks like this:
from openai import OpenAI
client = OpenAI(api_key="sk-...")
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a concise helper."},
{"role": "user", "content": "Summarize the incident report."}
],
temperature=0.2,
)
print(resp.choices[0].message.content)The critical observation: content is a plain string. That single-field assumption is baked into serializers, logging, and retry logic. Vision support does not change the endpoint path, authentication, or response shape. It only changes the allowed type of content from string to array of content parts.
If you are on a raw requests call or a framework like LangChain, the same rule applies—find where the user message string is constructed and stop treating it as scalar.
Step 2: Select a vision-capable model and endpoint
You need a model that accepts images. As of this writing, strong public options include gpt-4o, claude-3-5-sonnet, and llama-3.2-11b-vision. Your existing model parameter is the only required switch if your endpoint already routes to that provider.
If you would rather not hardcode provider URLs or maintain fallback logic yourself, an OpenAI-compatible gateway such as n4n.ai exposes one endpoint that addresses 240+ models and automatically falls back when a provider is rate-limited or degraded. You keep the same SDK and just change base_url:
client = OpenAI(
api_key="your-key",
base_url="https://api.n4n.ai/v1", # single endpoint, 240+ models
)Either way, the integration target stays the /chat/completions route. Do not stand up a second microservice for vision; the marginal complexity is not justified when the API contract is identical.
Step 3: Reshape the message content array
Vision models expect content to be an array of typed parts. The two relevant types are text and image_url. Here is the minimal JSON diff from the string form:
{
"role": "user",
"content": [
{ "type": "text", "text": "What is the license plate in this photo?" },
{ "type": "image_url", "image_url": { "url": "https://example.com/car.jpg" } }
]
}If you previously sent content as a string, wrap it in a text part. Do not mix string and array formats in the same message; the API will reject it with a validation error. You can place multiple image_url parts in one message to compare diagrams or process a batch of screenshots. Order matters for some models—put the instruction text part first so the model knows what to do with the images that follow.
Detail parameter
Many endpoints accept an optional detail field: "image_url": {"url": "...", "detail": "low"}. Use "low" for thumbnails when you only need coarse understanding; it cuts token cost dramatically.
Step 4: Send images by URL or base64
Remote URLs are simplest, but most backend pipelines generate images in memory—PDF renders, canvas snapshots, uploaded avatars. Use a data URI for base64:
import base64
def b64_image(path):
with open(path, "rb") as f:
return "data:image/png;base64," + base64.b64encode(f.read()).decode()
resp = client.chat.completions.create(
model="gpt-4o",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Extract the table rows."},
{"type": "image_url", "image_url": {"url": b64_image("scan.png")}},
],
}
],
)Keep the MIME prefix correct (image/png, image/jpeg). Base64 inflates payload size by ~33%; if your gateway meters egress, downscale to 1024px on the long edge before encoding unless you need OCR at full resolution. Streaming works unchanged—pass stream=True and iterate as you already do.
Step 5: Respect provider limits and error modes
Vision endpoints enforce constraints your text path never hit:
- Max image count per message (often 1–20).
- Max dimension or pixel budget (e.g., 2048px on the short side).
- Supported formats (PNG, JPEG, WEBP; GIF only on select models).
Wrap the call so you can catch 400 with image_too_large or invalid_image_format. Example:
from openai import APIError
try:
resp = client.chat.completions.create(...)
except APIError as e:
if "image" in e.message.lower():
# fallback to text-only or shrink image
resp = client.chat.completions.create(model="gpt-4o-mini", messages=[...])When you integrate vision models api across multiple providers, encode these limits in a config map keyed by model name. A naive try/except that retries the identical payload wastes tokens and latency.
Step 6: Keep your response parsing unchanged
The completion object is identical to text-only calls. You still read resp.choices[0].message.content. Usage may include vision tokens under resp.usage:
print(resp.usage.prompt_tokens, resp.usage.completion_tokens)Some gateways report prompt_tokens_details.image_tokens. Log it separately. Per-token metering means a silent image blow-up shows up here first—a 4K screenshot at high detail can cost 10x a text page. If you stream, the final usage arrives in the terminating chunk; buffer it.
Step 7: Add routing hints and cache control
When you sit behind a gateway, you can pass routing directives without vendor lock-in. OpenRouter-class proxies honor headers like x-router-prefer or body fields. Forward provider cache-control hints if your images are static:
resp = client.chat.completions.create(
model="claude-3-5-sonnet",
messages=[...],
extra_headers={"x-cache-control": "max-age=3600"},
)This tells the upstream to reuse preprocessed image features for repeated calls. Your existing text cache logic can stay; just extend the header pass-through. If your client already sends cache-control: max-age for prompts, confirm the gateway forwards it to the provider rather than stripping it.
Step 8: Verify the integration
Run a minimal end-to-end test with a known image. A curl smoke test:
curl https://api.openai.com/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Does this show a cat? Answer yes/no."},
{"type": "image_url", "image_url": {"url": "https://placekitten.com/200/200"}}
]
}]
}'Success criteria:
- HTTP 200 with a JSON body.
choices[0].message.contentcontains a coherent answer (“yes” for that URL).usage.prompt_tokensis greater than the text-only equivalent, confirming image tokens were counted.
For automated verification, drop this into a pytest:
def test_vision_call(client):
r = client.chat.completions.create(
model="gpt-4o",
messages=[{"role":"user","content":[
{"type":"text","text":"Color of the ball?"},
{"type":"image_url","image_url":{"url":"https://example.com/red.png"}}
]}],
)
assert "red" in r.choices[0].message.content.lower()
assert r.usage.prompt_tokens > 10If you use the gateway approach, swap the URL and key. The same test validates fallback: temporarily configure an invalid primary route and confirm the call still returns 200 via the secondary.
Production notes
Integrate vision models api by treating images as just another content part, not a new service. Keep your model list in env config, cap image size at the edge, and log image_tokens separately. Once the payload shape is correct, the rest is the same chat completion loop you already run—same retries, same timeout budgets, same observability. The only new discipline is watching image-specific limits and token blow-up.