Most pipelines upload an image, generate a caption, then feed that text into a separate function-calling step. A vision plus tool calling single API request collapses that into one round trip: the model sees the pixels and emits a tool call in the same response, which shrinks latency and removes error-prone glue code.
Step 1: Select a model that supports interleaved vision and tools
Not every multimodal model can reason over an image and return a structured tool call in the same completion. GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, and Gemini 1.5 Pro handle this natively. Many open-weight models behind inference gateways do not, so verify before wiring it into production.
If you are using a unified gateway, you can list models and filter client-side:
from openai import OpenAI
client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")
supported = []
for m in client.models.list().data:
# Heuristic: known prefixes that support vision+tools as of 2024
if any(tag in m.id for tag in ("gpt-4o", "claude-3-5", "gemini-1-5")):
supported.append(m.id)
print(supported)A gateway like n4n.ai exposes 240+ models behind one OpenAI-compatible endpoint, so the same filter works without per-provider SDKs. Pick one ID and pin it.
Step 2: Define the tool and send the image in one message
The OpenAI Chat Completions schema accepts a content array mixing text and image_url blocks. Tools are passed as a top-level tools list. The model decides whether to call the tool based on the image and prompt.
tools = [
{
"type": "function",
"function": {
"name": "crop_image",
"description": "Crop the input image to the given bounding box in normalized coordinates",
"parameters": {
"type": "object",
"properties": {
"x": {"type": "number", "description": "left edge 0-1"},
"y": {"type": "number", "description": "top edge 0-1"},
"w": {"type": "number", "description": "width 0-1"},
"h": {"type": "number", "description": "height 0-1"},
},
"required": ["x", "y", "w", "h"],
},
},
}
]
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Find the red truck and crop it tightly."},
{"type": "image_url", "image_url": {"url": "https://example.com/scene.jpg"}},
],
}
]
resp = client.chat.completions.create(
model="gpt-4o",
messages=messages,
tools=tools,
tool_choice="auto",
)
msg = resp.choices[0].message
print(msg.tool_calls)This is the core of a vision plus tool calling single API request: one create call, one image, one tool schema.
Step 3: Execute the returned tool call
The model may return tool_calls attached to the assistant message. You must parse the arguments (they arrive as a JSON string) and run your local function. Do not assume the coordinates are sane—validate them.
import json
if msg.tool_calls:
call = msg.tool_calls[0]
args = json.loads(call.function.arguments)
# Clamp to valid range
x, y, w, h = (max(0.0, min(1.0, args[k])) for k in ("x", "y", "w", "h"))
print(f"Model requested crop: {x},{y},{w},{h}")
# Your actual crop logic here
tool_result = f"cropped image saved at /tmp/crop_{call.id}.png"
else:
print("No tool call; model responded with text:", msg.content)If the model skipped the tool, it usually means the prompt was ambiguous or the image lacked the target object. Treat that as a normal branch, not an error.
Step 4: Feed the tool result back for a final answer
Append the assistant message (with tool_calls) and a tool role message, then call again. The model now produces a natural-language summary that references the executed action.
messages.append(msg) # assistant message with tool_calls
messages.append({
"role": "tool",
"tool_call_id": call.id,
"content": tool_result,
})
final = client.chat.completions.create(
model="gpt-4o",
messages=messages,
)
print(final.choices[0].message.content)This two-phase pattern is still fewer round trips than caption-then-call, and the first phase is the vision plus tool calling single API request that does the heavy lifting.
Step 5: Verify the workflow end to end
You need a deterministic check that the tool was invoked with coordinates inside the image and that the final text acknowledges the crop. A minimal verification script:
def test_vision_tool_flow():
# Using a fixed test image with a known red truck at normalized box 0.2,0.3,0.4,0.2
resp = client.chat.completions.create(
model="gpt-4o",
messages=[{"role":"user","content":[
{"type":"text","text":"Crop the red truck."},
{"type":"image_url","image_url":{"url":TEST_IMAGE}}]}],
tools=tools,
)
tc = resp.choices[0].message.tool_calls
assert tc is not None, "Expected a tool call"
a = json.loads(tc[0].function.arguments)
assert 0 <= a["x"] < a["x"]+a["w"] <= 1, "Invalid bbox"
assert 0 <= a["y"] < a["y"]+a["h"] <= 1, "Invalid bbox"
# Second call confirms acknowledgement
msgs = [resp.choices[0].message, {"role":"tool","tool_call_id":tc[0].id,"content":"ok"}]
out = client.chat.completions.create(model="gpt-4o", messages=msgs)
assert "crop" in out.choices[0].message.content.lower()
print("PASS")Run it against your chosen model. If it fails, inspect the raw arguments string—malformed JSON is rare but happens with smaller models.
How naive gateways force extra hops
Some providers expose vision only on a /v1/vision endpoint and tools only on /v1/chat. To combine them you must: (1) call vision to get a description, (2) inject that into a second chat call with tools, (3) handle the tool, (4) possibly call again. That quadruples tail latency and introduces transcription loss—the model describing the image may omit the precise coordinate the tool needs.
A single OpenAI-compatible route that accepts mixed content and tools removes that fragmentation. You write one request shape, and the gateway forwards it to any backend that honors it. This is the differentiator in multimodal routing: not just “supports images” and “supports functions” separately, but interleaved in one context window.
Pitfalls and production notes
Image token cost. High-resolution images burn tokens fast. Downscale before encoding if the task is coarse (e.g., “is there a truck?”). The tool call will still be precise enough.
Cache control. If you send the same image across multiple tool iterations, set cache_control on the image block where the provider supports it. Gateways that forward provider cache hints will reduce repeat cost on the second pass of Step 4.
Fallback behavior. If your pinned model is rate-limited, a gateway with automatic fallback can reroute to a peer model that also supports vision plus tool calling single API request. Your code should still validate the response shape, because argument naming can drift between vendors.
Tool choice rigidity. tool_choice="auto" lets the model abstain. If the image clearly contains the target, consider tool_choice={"type":"function","function":{"name":"crop_image"}} to force the call and avoid a text-only reply that wastes a round trip.
Bounding box conventions. Normalize coordinates (0–1) to avoid resolution coupling. If your tool expects pixels, convert after validation.
Closing verification checklist
- Model ID confirmed to support interleaved vision and tools
- One
createcall sends image + tool schema -
tool_callsparsed and arguments clamped - Tool result fed back in
role: toolmessage - Final completion references the action
- Automated test asserts bbox validity and acknowledgement
Follow those steps and you have a robust vision plus tool calling single API request path that beats the caption-then-call anti-pattern.