Building a multimodal agent with Claude Opus 4.5

To build multimodal agent with Claude Opus 4.5 you need more than a chat wrapper. The model ingests images and emits structured tool calls, so the real engineering is the control loop that executes those calls as screen actions and feeds observations back.

Step 1: Set up API access and dependencies

Install the client and screen-control libraries:

pip install openai pyautogui pillow

Point the OpenAI client at an endpoint that serves Claude Opus 4.5. If you route through n4n.ai, a single OpenAI-compatible endpoint exposes that model alongside 240+ others and handles provider degradation with automatic fallback.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",  # OpenAI-compatible gateway
    api_key="YOUR_KEY",
)

MODEL = "anthropic/claude-opus-4.5"  # example routing name

Keep the model name aligned with your gateway’s catalog. The rest of the loop is model-agnostic as long as the endpoint speaks Chat Completions.

Step 2: Define the agent’s action tools

When you build multimodal agent with Claude Opus 4.5, tool definitions determine whether the model selects actions correctly. Define a minimal computer-use surface: screenshot, click, type.

tools = [
    {
        "type": "function",
        "function": {
            "name": "screenshot",
            "description": "Capture the current screen and return it as an image.",
            "parameters": {"type": "object", "properties": {}},
        },
    },
    {
        "type": "function",
        "function": {
            "name": "click",
            "description": "Click at pixel coordinates.",
            "parameters": {
                "type": "object",
                "properties": {
                    "x": {"type": "integer"},
                    "y": {"type": "integer"},
                },
                "required": ["x", "y"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "type_text",
            "description": "Type a string into the focused element.",
            "parameters": {
                "type": "object",
                "properties": {"text": {"type": "string"}},
                "required": ["text"],
            },
        },
    },
]

You can extend this with key_press or scroll later. Keep the schema tight; verbose descriptions degrade tool-selection accuracy.

Step 3: Capture and encode screen state

The agent needs to see the screen. Grab a screenshot, encode as PNG base64, and wrap it in a content block.

import base64
import io
from PIL import Image
import pyautogui

def capture_screen():
    img = pyautogui.screenshot()
    buf = io.BytesIO()
    img.save(buf, format="PNG")
    b64 = base64.b64encode(buf.getvalue()).decode()
    return f"data:image/png;base64,{b64}"

Never send raw file paths to a remote model unless your gateway is on the same host. Inline data URLs keep the loop self-contained.

Step 4: Implement the agent loop

The loop sends the task, the latest screenshot, and tool results. It stops when the model returns text without tool calls. This structure is the core of any multimodal agent with Claude Opus 4.5: observe, act, append, repeat.

def run_agent(task, max_steps=10):
    messages = [
        {"role": "system", "content": "You are a computer-use agent. Use tools to complete the task."},
        {"role": "user", "content": task},
    ]

    for step in range(max_steps):
        screen = capture_screen()
        messages.append({
            "role": "user",
            "content": [
                {"type": "text", "text": "Current screen:"},
                {"type": "image_url", "image_url": {"url": screen}},
            ],
        })

        resp = client.chat.completions.create(
            model=MODEL,
            messages=messages,
            tools=tools,
            tool_choice="auto",
        )
        msg = resp.choices[0].message
        messages.append(msg)

        if not msg.tool_calls:
            return msg.content

        for call in msg.tool_calls:
            result = execute_tool(call)
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": result,
            })
    return "Agent exceeded step budget"

Step 5: Execute computer actions safely

Map tool calls to pyautogui. Guard coordinates against screen bounds to avoid crashes.

import json
import pyautogui

def execute_tool(call):
    name = call.function.name
    args = json.loads(call.function.arguments)
    if name == "screenshot":
        return "captured"
    elif name == "click":
        w, h = pyautogui.size()
        x = min(max(0, args["x"]), w - 1)
        y = min(max(0, args["y"]), h - 1)
        pyautogui.click(x, y)
        return f"clicked {x},{y}"
    elif name == "type_text":
        pyautogui.write(args["text"], interval=0.05)
        return f"typed {len(args['text'])} chars"
    else:
        return "unknown tool"

Run the agent on a sandboxed machine. Computer-use scripts will move the real mouse and keyboard.

Step 6: Run a concrete task and verify

The fastest way to build multimodal agent with Claude Opus 4.5 that reliably acts is to start with a narrow task. Task: open the system calculator, compute 2+2, and report the result.

result = run_agent("Open the calculator app, type 2+2, and tell me the displayed result.")
print(result)

Verification: before trusting the agent, assert side effects. After the run, capture a final screenshot and OCR it, or check the calculator process state. A simple check:

final_screen = capture_screen()
# use an OCR lib or visual diff to confirm "4" is present
assert "4" in ocr_text(final_screen), "Agent did not produce expected output"

If the loop returns the correct value and the screen shows 4, the build works. If it stalls, increase max_steps or tighten the system prompt.

Step 7: Cache static context and meter usage

System prompts and tool schemas rarely change mid-run. Mark them cacheable so the gateway can reuse prompt prefixes. n4n.ai forwards provider cache-control hints, so you can set cache_control on the system block if your endpoint supports it.

messages[0]["content"] = [
    {
        "type": "text",
        "text": "You are a computer-use agent. Use tools to complete the task.",
        "cache_control": {"type": "ephemeral"},
    }
]

Per-token metering lets you track cost per task. Log resp.usage after each call:

print(resp.usage.prompt_tokens, resp.usage.completion_tokens)

Production considerations

Computer-use agents fail on unexpected dialogs. Add a watchdog that resets the session if no tool call appears for three steps. Handle rate limits with exponential backoff; the gateway’s fallback covers provider-side throttling but client-side retries still matter.

Keep the screenshot resolution modest (e.g., 1280x720) to limit token burn. Claude Opus 4.5 processes images as tiles, so smaller captures reduce latency without losing UI legibility.

To build multimodal agent with Claude Opus 4.5 at scale, separate the policy model from the executor: run the loop in a worker, stream screens to a broker, and keep human-in-the-loop approval for destructive actions.