A GPT-4o UI testing agent turns a browser screenshot into actionable test steps without hand-written selectors. This tutorial builds one with Playwright and the OpenAI vision API, driving a real login flow from pixels alone.
Prerequisites
- Python 3.11 or newer
- Playwright for browser control:
pip install playwright && playwright install chromium - OpenAI Python client:
pip install openai - An OpenAI API key (or any OpenAI-compatible endpoint) exported as
OPENAI_API_KEY tenacityfor retries:pip install tenacity
You should be comfortable with async Python and basic CSS layouts. The GPT-4o UI testing agent we build will not use DOM selectors for reasoning; it will only see rendered pixels and issue coordinate-based actions.
Architecture: screenshot, reason, act
The loop is three steps:
- Capture the current viewport as a PNG.
- Send the image to GPT-4o with a strict JSON schema describing allowed actions.
- Parse the response and execute the actions via Playwright’s input APIs.
Repeat until the model reports done or a max step count is hit. This keeps the GPT-4o UI testing agent grounded in the actual rendered UI rather than brittle selectors that break on every redesign.
Step 1: Capture a screenshot
Use Playwright’s page.screenshot() and base64-encode it for the API. Keep the viewport fixed so coordinate math stays simple.
import asyncio, base64
from playwright.async_api import async_playwright
async def grab_screenshot(page):
png = await page.screenshot(type="png")
return base64.b64encode(png).decode("utf-8")Launch a browser and navigate to a target:
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch()
page = await browser.new_page(viewport={"width": 1280, "height": 800})
await page.goto("https://example.com/login")
b64 = await grab_screenshot(page)
print(len(b64), "bytes of base64 png")
await browser.close()
asyncio.run(main())Expected output at this checkpoint:
34211 bytes of base64 pngThat string is what we feed to the model. At 1280×800, a PNG compresses to roughly 25–40 KB; base64 adds ~33% overhead.
Step 2: Define the vision prompt and schema
GPT-4o needs a constrained output format. We force JSON via a system prompt and a clear action vocabulary. The model returns a list of actions; each action is either click, type, or done.
SYSTEM_PROMPT = """
You are a UI testing agent. Given a screenshot, output JSON with an "actions" array.
Each action is one of:
{"type": "click", "x": int, "y": int}
{"type": "type", "x": int, "y": int, "text": str}
{"type": "done", "reason": str}
Coordinates are relative to the 1280x800 screenshot. Do not guess DOM selectors.
If the page shows an error, retry the field. Stop only when the goal is met.
"""We send the image as a data URL in the user message. No few-shot examples are needed; GPT-4o handles the schema from the instruction alone.
Step 3: Call GPT-4o and parse the response
Use the standard OpenAI client. Set response_format to json_object to avoid free text. If you operate behind a gateway, set base_url accordingly.
from openai import OpenAI
client = OpenAI() # or OpenAI(base_url="https://your-endpoint/v1")
def reason(b64_image: str) -> dict:
resp = client.chat.completions.create(
model="gpt-4o",
response_format={"type": "json_object"},
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": [
{"type": "text", "text": "Current state of the login page."},
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64_image}"}}
]}
],
max_tokens=300
)
import json
return json.loads(resp.choices[0].message.content)Expected JSON from the model for a blank login form:
{
"actions": [
{"type": "click", "x": 420, "y": 250},
{"type": "type", "x": 420, "y": 250, "text": "testuser"}
]
}The model returns coordinates in screenshot space. Because we pinned the viewport to 1280×800, those map 1:1 to Playwright’s mouse.click.
Step 4: Execute actions in the browser
Map the parsed actions to Playwright calls. A type action first clicks to focus, then sends keystrokes.
async def execute_actions(page, actions):
for act in actions:
if act["type"] == "click":
await page.mouse.click(act["x"], act["y"])
elif act["type"] == "type":
await page.mouse.click(act["x"], act["y"])
await page.keyboard.type(act["text"], delay=20)
elif act["type"] == "done":
print("Agent done:", act.get("reason"))
return True
return FalseThe delay parameter mimics human input and avoids triggering anti-bot guards that filter instantaneous fills.
Step 5: Run a multi-step login test
Wire the pieces into a loop with a step limit. After each action batch, re-screenshot so the model sees the result.
async def run_test():
async with async_playwright() as p:
browser = await p.chromium.launch()
page = await browser.new_page(viewport={"width": 1280, "height": 800})
await page.goto("https://example.com/login")
for step in range(10):
b64 = await grab_screenshot(page)
plan = reason(b64)
done = await execute_actions(page, plan.get("actions", []))
if done:
break
title = await page.title()
print("Final title:", title)
await browser.close()
asyncio.run(run_test())A successful run prints:
Agent done: Login succeeded, dashboard visible
Final title: DashboardIf the model mistypes, the next screenshot reveals the error and it can correct course. That feedback loop is what makes a GPT-4o UI testing agent resilient to minor layout shifts.
Why coordinate-based actions instead of selectors
Traditional Playwright tests rely on text=, #id, or XPath. Those break when the frontend swaps frameworks or renames classes. Vision-based control trades that fragility for token cost. The agent reasons about what the user sees, not the DOM the developer wrote. For volatile UIs—marketing pages, A/B tests, LLM-generated interfaces—this is the only sustainable approach.
Calibrating the viewport
If you must run at a different resolution, scale coordinates:
def scale(act, shot_w, shot_h, view_w, view_h):
act["x"] = int(act["x"] * view_w / shot_w)
act["y"] = int(act["y"] * view_h / shot_h)
return actPass the screenshot dimensions (from the PNG header or a fixed assumption) and the live viewport. Keep both known constants to avoid rounding drift.
Handling rate limits and retries
Vision calls are heavier than text. In CI, a single flaky provider outage should not fail your suite. If you point the OpenAI client at an OpenAI-compatible gateway like n4n.ai, you get automatic fallback across providers when GPT-4o is rate-limited, plus per-token metering. The endpoint honors your routing directives and forwards cache-control hints, which matters when you replay the same screenshot across retries.
For local runs, wrap reason() with tenacity:
from tenacity import retry, stop_after_attempt, wait_fixed
@retry(stop=stop_after_attempt(3), wait=wait_fixed(2))
def reason_with_retry(b64):
return reason(b64)Debugging the agent
When the agent loops or clicks the wrong spot, dump every screenshot to disk with a step index:
await page.screenshot(path=f"debug/step_{step}.png")Inspect the images alongside the JSON plans. Common failure modes:
- Model clicks a disabled button because it looks enabled. Add a rule: “If a button is greyed, do not click; report blocked.”
- Text typed into the wrong field because two inputs overlap visually. Tighten the prompt to require field labels.
Token cost and latency notes
A 1280×800 PNG sent as base64 consumes roughly 1.5–2K tokens of image input per call under GPT-4o’s tiling scheme. A ten-step test therefore costs ~20K input tokens plus small output. Latency per step is dominated by network and model inference, typically 1–3 seconds. Parallelize independent test paths across browser contexts if you need speed.
Closing notes
The agent described here is deliberately minimal. For production, add:
- A diff check between screenshots to detect idle loops.
- Schema validation with pydantic before execution.
- A fallback to DOM selectors when the model returns low-confidence coordinates.
Building a GPT-4o UI testing agent this way trades selector maintenance for token cost, and for volatile UIs that trade is often worth it.