How to send image and text requests to vision LLMs in Python

A python multimodal request vision llm combines a text prompt with one or more images in a single chat completion call. Most providers expose an OpenAI-compatible /v1/chat/completions endpoint, so you can use plain requests or httpx without SDK lock-in.

Step 1: Choose your HTTP client and endpoint

Use requests if your code is synchronous and simple. Use httpx when you need async concurrency or HTTP/2. Both speak JSON over HTTPS and require an Authorization: Bearer <key> header.

Point the base URL at your provider or gateway. The OpenAI-compatible contract is stable across vendors:

BASE_URL = "https://api.openai.com/v1"  # or your gateway's URL
API_KEY = "sk-..."  # load from env in real code

HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json",
}

If you route through an OpenAI-compatible gateway like n4n.ai, the same payload triggers automatic fallback across providers and per-token metering, but your client code stays identical.

Step 2: Prepare the image input

Vision models accept images either as a public URL or as a base64 data URI. URLs are cheaper to send; base64 works for local files or private blobs.

For a local file, encode it:

import base64

def encode_image(path: str) -> str:
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

data_uri = f"data:image/jpeg;base64,{encode_image('receipt.jpg')}"

For a remote image, just use the URL:

image_url = "https://example.com/cat.png"

Keep detail set to "auto" unless you specifically need "low" to cut tokens on large images.

Step 3: Build the multimodal message array

The messages field takes a content array. Each element is either a text part or an image_url part. Order matters: lead with the instruction, then attach images.

payload = {
    "model": "gpt-4o-mini",  # or any vision-capable model
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Extract the total amount from this receipt.",
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": data_uri,
                        "detail": "auto",
                    },
                },
            ],
        }
    ],
    "max_tokens": 300,
}

This structure is the core of any python multimodal request vision llm call. You can add multiple image_url parts in the same message if the model supports it.

Step 4: Send the request

Synchronous with requests:

import requests

resp = requests.post(
    f"{BASE_URL}/chat/completions",
    headers=HEADERS,
    json=payload,
    timeout=30,
)

Asynchronous with httpx:

import httpx

async def call_vision():
    async with httpx.AsyncClient(timeout=30) as client:
        resp = await client.post(
            f"{BASE_URL}/chat/completions",
            headers=HEADERS,
            json=payload,
        )
    return resp

Set timeout explicitly. Vision calls lag behind text-only ones because image pre-processing adds latency.

Step 5: Parse and handle errors

Check the status code before touching the JSON. A 200 returns choices[0].message.content. A 429 or 5xx means you should retry or fall back.

if resp.status_code != 200:
    # log resp.text, maybe exponential backoff
    raise RuntimeError(f"Vision call failed: {resp.status_code} {resp.text}")

data = resp.json()
answer = data["choices"][0]["message"]["content"]
usage = data.get("usage", {})

When a provider is rate-limited, the python multimodal request vision llm pattern lets you swap model or re-post to a fallback endpoint without changing the body. Gateways that honor client routing directives will do this automatically if you pass the right header.

Step 6: Verify success

Verification is more than a 200. Assert the model returned structured content and that token usage looks sane.

assert answer.strip(), "Empty response from model"
assert usage.get("total_tokens", 0) > 0, "No tokens metered"

print("MODEL ANSWER:", answer)
print("TOKENS USED:", usage)

Run the script against a known image. For a receipt, confirm the extracted total matches the visible number. If the assertion fails, inspect resp.text and the detail level on your image parts.

Streaming the response

For long captions, stream to avoid blocking:

payload["stream"] = True
with requests.post(f"{BASE_URL}/chat/completions", headers=HEADERS, json=payload, stream=True) as r:
    for line in r.iter_lines():
        if line and line.startswith(b"data:"):
            chunk = line[5:].strip()
            if chunk != b"[DONE]":
                print(chunk.decode())

The SSE format returns delta text fragments; concatenate them client-side.

Forwarding cache-control hints

Some providers cache prompt prefixes. If your gateway forwards cache-control hints, add cache_control to a text part:

{
    "type": "text",
    "text": "System context: invoice schema v2.",
    "cache_control": {"type": "ephemeral"},
}

This cuts cost on repeated document types. Not all models honor it; test with a usage report.

Why raw REST beats SDKs here

SDKs abstract the wire format and sometimes block unknown parameters like cache_control. Raw requests or httpx lets you send exactly what the spec allows and debug the raw JSON. You also avoid dependency bloat in lambda functions.

The python multimodal request vision llm workflow above is portable across every OpenAI-compatible endpoint. Write the payload once, ship it behind a thin client, and swap models by changing a string.