GPT-4o vision agents that read screenshots and click

Building a GPT-4o vision agent screenshot automation loop means solving two coupled problems: streaming a faithful image of the screen into the model, and converting its spatial reasoning back into a precise mouse event. This guide implements a working capture–infer–act cycle using the OpenAI-compatible chat completions API and a few lines of Python.

Step 1: Capture a clean screenshot

Start with a deterministic capture routine. pyautogui works, but mss is faster and avoids compositor lag on Linux. Grab the primary monitor and encode as PNG bytes.

import mss
import mss.tools

def grab_screen():
    with mss.mss() as sct:
        monitor = sct.monitors[1]  # primary
        shot = sct.grab(monitor)
        return mss.tools.to_png(shot.rgb, shot.size)

Keep the raw bytes. Base64-encode only at request time to avoid holding two copies in memory.

If you run on a HiDPI display, monitor width may be in logical pixels while the model sees the raw raster. Pass the actual pixel dimensions alongside the image so the agent can map coordinates later.

Step 2: Send the image to GPT-4o with a constrained prompt

The model needs explicit instructions: return a JSON object, no commentary. Use JSON mode to force structure.

import base64
from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",  # OpenAI-compatible, 240+ models, fallback on degrade
    api_key="YOUR_KEY"
)

def infer_action(png_bytes, screen_w, screen_h):
    b64 = base64.b64encode(png_bytes).decode()
    data_uri = f"data:image/png;base64,{b64}"
    resp = client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": (
                "You are a UI agent. Output JSON with keys: "
                "action (click|none), x (int abs pixel), y (int abs pixel), "
                "reason (short string). Use the provided screen size."
            )},
            {"role": "user", "content": [
                {"type": "text", "text": f"Screen {screen_w}x{screen_h}. Click the 'Submit' button."},
                {"type": "image_url", "image_url": {"url": data_uri}}
            ]}
        ],
        max_tokens=200
    )
    return resp.choices[0].message.content

This GPT-4o vision agent screenshot automation call returns a string like {"action":"click","x":842,"y":430,"reason":"blue button"}. The gateway forwards provider cache-control hints, so repeated static UI regions cost less if you set cache_control on the image block (supported by some providers; check docs).

Step 3: Parse and validate the response

Never trust the model’s types. Load JSON and clamp coordinates.

import json

def parse_action(raw, screen_w, screen_h):
    try:
        obj = json.loads(raw)
    except json.JSONDecodeError:
        return None
    if obj.get("action") != "click":
        return None
    x = int(obj["x"]); y = int(obj["y"])
    if not (0 <= x <= screen_w and 0 <= y <= screen_h):
        return None
    return x, y

If validation fails, fall back to a no-op or a human-in-the-loop prompt. Do not retry blindly; GPT-4o occasionally returns out-of-bounds values on occluded UI.

Step 4: Map coordinates and click

pyautogui uses the same coordinate space as mss on single-monitor setups. Click with a small pause to let the OS process the event.

import pyautogui

def act(x, y):
    pyautogui.moveTo(x, y, duration=0.2)
    pyautogui.click()

For multi-monitor, mss monitor offsets matter. Subtract the monitor’s left/top from the model’s coordinates if you captured a non-primary monitor.

Safety: set pyautogui.FAILSAFE = True. Yank the mouse to a corner to abort during testing.

Step 5: Verify the action succeeded

A click with no verification is a guess. Capture a second screenshot 500ms later and ask the model whether the expected state changed.

import time

def verify(png_before, png_after, expectation):
    b64_before = base64.b64encode(png_before).decode()
    b64_after = base64.b64encode(png_after).decode()
    resp = client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": "Output JSON {changed: bool, note: string}"},
            {"role": "user", "content": [
                {"type": "text", "text": f"Did this happen: {expectation}?"},
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64_before}"}},
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64_after}"}}
            ]}
        ]
    )
    return json.loads(resp.choices[0].message.content).get("changed")

If changed is false, log the pair for debugging. This closes the GPT-4o vision agent screenshot automation loop with real feedback.

Step 6: Run the agent loop

Wire the pieces together. Limit iterations to avoid infinite loops.

def run_agent(task, max_steps=10):
    for step in range(max_steps):
        before = grab_screen()
        w, h = pyautogui.size()
        raw = infer_action(before, w, h)
        coords = parse_action(raw, w, h)
        if not coords:
            print("no action", raw)
            continue
        act(*coords)
        time.sleep(0.5)
        after = grab_screen()
        if verify(before, after, task):
            print("task done")
            return True
    return False

Run run_agent("submit the form"). You should see the cursor move and click the button, then the loop exit.

Step 7: Harden for production

The toy loop above breaks under real conditions. Address these:

Timeouts and retries

Wrap infer_action in a ten-second timeout. GPT-4o calls over a gateway can hang if a provider is degraded. If you use a routing gateway like n4n.ai, it auto-falls back to a healthy provider, but your client still needs a socket timeout.

from requests.exceptions import Timeout
try:
    raw = infer_action(before, w, h)
except Timeout:
    continue

Token metering

Vision calls burn tokens on image size. Track resp.usage.total_tokens per step. Per-token metering at the gateway lets you cap spend per session.

Prompt caching

If the UI chrome is static, mark the screenshot with cache_control: {"type": "ephemeral"} where the API supports it. The gateway forwards the hint, reducing repeat cost on unchanged frames.

Determinism

Set temperature=0 for action steps. You want the same button clicked every time, not a creative variant.

Verify success end to end

Success criteria: the agent completes the stated task without manual intervention across three runs on a fresh browser session. Measure by asserting run_agent returns True and the target DOM element (or pixel region) reflects the post-click state. For a submit button, check that the form confirmation appears in the next screenshot.

If the agent misses, increase screenshot contrast or crop to the relevant region before sending. GPT-4o reads full-screen 1080p fine, but small 12px labels need zoom.

That’s the entire pipeline. Capture, infer with constrained JSON, validate, click, verify. Build from here: add keyboard actions, region-based cropping, and a replay buffer for self-correction.