Sending images to a vision model with curl

A curl vision model image request is the most direct way to probe a multimodal endpoint without dragging in a SDK or writing a Python client. Below is the exact sequence to encode an image, assemble the payload, and POST it to any OpenAI-compatible vision model, including gateways that proxy multiple providers.

Step 1: Set up credentials and choose a model

Every inference gateway expects a bearer token. Export it so it never lands in your shell history as a literal:

export OPENAI_API_KEY="sk-..."   # or your gateway token
export BASE_URL="https://api.openai.com/v1"

If you route through n4n.ai, the base URL is https://api.n4n.ai/v1 and the same OpenAI-compatible schema works across 240+ models with automatic fallback when a provider is degraded. The model string may need a prefix (e.g., openai/gpt-4o) depending on the gateway’s routing rules.

Pick a vision-capable model. Known working OpenAI-compatible identifiers include gpt-4o, gpt-4-vision-preview, and various Claude 3 variants behind a proxy. Check the provider’s model list before sending a large image.

Step 2: Prepare the image

You have two options: pass a publicly reachable URL or inline the bytes as base64. URLs are simpler but leak the asset to the provider’s egress and may be blocked by private networks. Base64 keeps the image local until the TLS post, but inflates payload size by ~33%.

Option A: Remote URL

Host the file on a server you control or an S3 bucket with a signed link. The JSON later references it directly.

Option B: Base64 encode a local file

On Linux/macOS:

# -w0 prevents line wraps that break JSON
base64 -w0 photo.jpg > photo.b64
IMG_B64=$(cat photo.b64)

On macOS base64 lacks -w; use base64 photo.jpg | tr -d '\n' instead. Keep the original format—JPEG and PNG are universally accepted; TIFF or HEIC often are not.

Step 3: Construct the request payload

The OpenAI chat completions schema accepts a content array mixing text and image blocks. Write a minimal JSON file:

{
  "model": "gpt-4o",
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "Describe this image in one sentence." },
        { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,__B64__" } }
      ]
    }
  ],
  "max_tokens": 300
}

Replace __B64__ with the base64 string. For a URL, set "url": "https://example.com/photo.jpg" instead of the data URI.

To avoid hand-editing, generate the payload with jq:

jq -n \
  --arg b64 "$IMG_B64" \
  --arg model "gpt-4o" \
  '{
    model: $model,
    messages: [
      { role: "user",
        content: [
          { type: "text", text: "Describe this image in one sentence." },
          { type: "image_url", image_url: { url: ("data:image/jpeg;base64," + $b64) } }
        ]
      }
    ],
    max_tokens: 300
  }' > req.json

This produces a valid req.json with no trailing newline issues.

Step 4: Send the curl vision model image request

Now POST it. Use --fail-with-body so curl exits non-zero on HTTP errors but still prints the response:

curl -sS --fail-with-body "$BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d @req.json > resp.json

If you prefer a one-liner without a temp file, pipe the jq output:

jq -n --arg b64 "$IMG_B64" '{model:"gpt-4o",messages:[{role:"user",content:[{type:"text",text:"What is in this image?"},{type:"image_url",image_url:{url:("data:image/jpeg;base64,"+$b64)}}]}]}' | \
curl -sS "$BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d @-

The -d @- reads the body from stdin. This is the core curl vision model image request pattern you will reuse.

Step 5: Verify the response

A successful call returns HTTP 200 and a JSON object with a choices array. Inspect it:

jq '.choices[0].message.content' resp.json

Expected output is a string containing the model’s description. If you got null, check the full response:

jq '.' resp.json

A well-formed success looks like:

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "model": "gpt-4o",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "A red bicycle leaning against a brick wall."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 1125, "completion_tokens": 12, "total_tokens": 1137 }
}

The usage block confirms the image was tokenized (prompt_tokens will be higher than the text alone). That is your proof the image reached the model.

Step 6: Handle errors and edge cases

Vision requests fail in predictable ways:

• 401 Unauthorized – token missing or malformed. Echo $OPENAI_API_KEY length (not value) to debug.
• 400 Bad Request – usually malformed base64 or unsupported content shape. Validate JSON with jq empty req.json.
• 413 Payload Too Large – image exceeds provider limit. Downscale with ffmpeg or sips before encoding.
• 429 Too Many Requests – rate limit. Implement a short backoff. Gateways with fallback may auto-retry on a different provider; if you set X-Route-Directive headers, honor them.

When sending through a gateway that honors client routing directives, you can pin a provider or force cache behavior:

curl -sS "$BASE_URL/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-Cache-Control: no-store" \
  -d @req.json

The X-Cache-Control hint forwards provider cache-control intent and prevents prompt caching if your image is ephemeral.

Step 7: Script it for repeatable tests

For local experimentation, wrap the steps in a function:

vision_req() {
  local img="$1" prompt="${2:-Describe this image.}"
  local b64
  b64=$(base64 -w0 "$img" 2>/dev/null || base64 "$img" | tr -d '\n')
  jq -n --arg b64 "$b64" --arg p "$prompt" '{
    model: "gpt-4o",
    messages: [ { role: "user", content: [
      { type: "text", text: $p },
      { type: "image_url", image_url: { url: ("data:image/jpeg;base64,"+$b64) } }
    ] } ],
    max_tokens: 200
  }' | curl -sS "$BASE_URL/chat/completions" \
    -H "Authorization: Bearer $OPENAI_API_KEY" \
    -H "Content-Type: application/json" -d @- | jq -r '.choices[0].message.content'
}

Run vision_req photo.jpg "What color is the car?" and get plain text back. This eliminates repeated boilerplate and makes the curl vision model image request workflow trivial to invoke from CI or a quick shell session.

Notes on token cost and latency

Images are encoded into tokens via a patch grid (e.g., 512×512 tiles). A 1024×1024 PNG may consume ~1000+ prompt tokens before any text. If you only need a yes/no answer, resize to 512px on the long edge first. Latency scales with image area and network round-trip; base64 inlining avoids a second HTTP fetch but increases upload size.

Stick to the OpenAI-compatible contract shown above and the same payload will work against virtually any modern multimodal endpoint. The curl vision model image request is not glamorous, but it is the fastest way to confirm a model sees what you think it sees.