Building a multimodal chat app with the OpenAI Vision API

Building an openai vision api chat app requires handling image bytes, not just text prompts. This tutorial walks from a single-shot vision query to a stateful FastAPI service, with runnable Python and expected outputs at each checkpoint.

Prerequisites

• Python 3.10 or newer
• openai Python package (v1.0.0+)
• fastapi and uvicorn for the HTTP layer
• pillow for image validation (optional but recommended)
• An OpenAI API key exported as OPENAI_API_KEY

pip install openai fastapi uvicorn pillow
export OPENAI_API_KEY="sk-..."

The multimodal message shape

OpenAI’s chat completions API accepts a content field that is either a string (text-only) or a list of typed parts. For vision, you mix text and image_url parts in one user message:

{
  "role": "user",
  "content": [
    {"type": "text", "text": "What is in this image?"},
    {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,...."}}
  ]
}

The image_url can be a public URL or a data URI. For a self-contained openai vision api chat app, base64 data URIs avoid server-side fetch complexity.

Step 1: Single image query

Encode a local file and send one question.

import base64
from openai import OpenAI

client = OpenAI()

def encode_image(path: str) -> str:
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_b64 = encode_image("receipt.jpg")
data_uri = f"data:image/jpeg;base64,{image_b64}"

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is the total amount on this receipt?"},
                {"type": "image_url", "image_url": {"url": data_uri}},
            ],
        }
    ],
    max_tokens=300,
)

print(response.choices[0].message.content)

Expected output (truncated):

The total amount on the receipt is $42.97.

The model returns text only. The image is consumed as input context and never echoed.

Step 2: Multi-turn conversation state

A real openai vision api chat app keeps a messages list across turns. You append user messages (text and/or image) and assistant responses.

from typing import Optional

messages = []

def ask(text: str, image_path: Optional[str] = None):
    content = [{"type": "text", "text": text}]
    if image_path:
        b64 = encode_image(image_path)
        content.append(
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}}
        )
    messages.append({"role": "user", "content": content})

    resp = client.chat.completions.create(model="gpt-4o", messages=messages, max_tokens=500)
    reply = resp.choices[0].message.content
    messages.append({"role": "assistant", "content": reply})
    return reply

# Turn 1: image + question
print(ask("Describe this diagram", "arch.png"))
# Turn 2: follow-up without image
print(ask("What would change if we added a cache layer?"))

The second turn works because the prior assistant message and the earlier user image remain in context. Be aware that image tokens count against context limits; repeated images across turns inflate usage. Trim history when you approach the model’s window.

Step 3: Expose it over HTTP

Wrap the logic in FastAPI to accept JSON. The client sends base64 to avoid server-side fetch surprises.

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()

class ChatRequest(BaseModel):
    text: str
    image_b64: Optional[str] = None
    history: list = []

@app.post("/chat")
def chat(req: ChatRequest):
    content = [{"type": "text", "text": req.text}]
    if req.image_b64:
        content.append(
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{req.image_b64}"}}
        )
    messages = req.history + [{"role": "user", "content": content}]
    try:
        resp = client.chat.completions.create(
            model="gpt-4o", messages=messages, max_tokens=500
        )
    except Exception as e:
        raise HTTPException(status_code=502, detail=str(e))
    return {"reply": resp.choices[0].message.content, "history": messages + [{"role": "assistant", "content": resp.choices[0].message.content}]}

Run with:

uvicorn main:app --port 8000

Test with curl:

curl -s localhost:8000/chat -H 'Content-Type: application/json' -d '{
  "text": "What color is the sky in this photo?",
  "image_b64": "'$(base64 -w0 sky.jpg)'"
}'

Expected JSON:

{
  "reply": "The sky is mostly blue with white clouds.",
  "history": [
    {"role": "user", "content": [{"type": "text", "text": "..."}, {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}]},
    {"role": "assistant", "content": "The sky is mostly blue with white clouds."}
  ]
}

Step 4: Stream tokens to the client

For chat UX, stream the assistant reply. OpenAI’s SDK supports stream=True.

from fastapi.responses import StreamingResponse

@app.post("/chat/stream")
def chat_stream(req: ChatRequest):
    content = [{"type": "text", "text": req.text}]
    if req.image_b64:
        content.append({"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{req.image_b64}"}})
    messages = req.history + [{"role": "user", "content": content}]
    stream = client.chat.completions.create(model="gpt-4o", messages=messages, max_tokens=500, stream=True)
    return StreamingResponse((chunk.choices[0].delta.content or "" for chunk in stream), media_type="text/plain")

The client receives incremental text. The image is sent only once per request; the stream carries deltas.

Handling provider degradation

If you deploy this openai vision api chat app to production, provider rate limits will eventually bite. Pointing the OpenAI client at an OpenAI-compatible gateway that adds automatic fallback shields you from single-provider outages. For example, n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models and automatically fails over when a provider is rate-limited or degraded, while still honoring your client routing directives and provider cache-control hints.

Swap the client base URL:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",
    api_key=os.environ["N4N_API_KEY"],
)

The rest of the code stays identical because the request shape matches the OpenAI spec.

Debugging common failures

429 Too Many Requests — Back off and retry. With a gateway, fallback may already redirect to a secondary provider.

400 Invalid image — Ensure the data URI prefix matches the actual bytes (image/png, image/jpeg). Corrupt base64 raises a validation error before the model is called.

Context length exceeded — Image tokens are expensive. Downscale with Pillow before encoding:

from PIL import Image
import io

def downscale(path: str, max_side: int = 1024) -> str:
    img = Image.open(path)
    img.thumbnail((max_side, max_side))
    buf = io.BytesIO()
    img.save(buf, format="JPEG")
    return base64.b64encode(buf.getvalue()).decode("utf-8")

Production notes

• Token metering: Use response.usage to log prompt_tokens and completion_tokens. If you use a gateway, per-token usage metering appears in the same field.
• History trimming: Drop older image turns when context exceeds model limits. Keep text summaries instead.
• Auth: Put the endpoint behind an API key or session check; do not expose unthrottled /chat to the internet.
• Caching: Forward cache_control hints if your provider supports prompt caching; gateways pass them through.

A minimal openai vision api chat app is now functional: it encodes images, maintains conversation, serves HTTP, and can fail over across providers. Extend with persistence and a frontend as needed.