A computer use shopping agent turns a vision-language model into a remote worker that clicks, types, and screenshots its way through an e-commerce site. In this tutorial we build one against Anthropic’s computer-use API, wiring it to a real browser so you can watch it log in, add items to a cart, and stop at checkout. The same pattern applies to any site where you’d rather not write brittle selectors.
Prerequisites
- Python 3.11 or newer
- An Anthropic API key exported as
ANTHROPIC_API_KEY - The
anthropicandplaywrightPython packages - Chromium binary installed for Playwright
pip install anthropic playwright
playwright install chromiumYou should be comfortable reading base64 screenshots and mapping pixel coordinates to mouse events. No frontend framework knowledge is required—the agent avoids DOM selectors entirely.
Architecture
We run a headless Chromium instance via Playwright. Each turn we capture a screenshot, send it to Claude with the computer_20241022 tool, and execute the returned action (click, type, scroll, key, wait, or finish) against the live page. The loop terminates when the model emits finish or we hit a step cap.
A computer use shopping agent needs a deterministic viewport. We use 1024×768 and strip animations so layout shifts don’t desync the model’s coordinate memory. As a test bed we use https://www.saucedemo.com, a public Sandbox with fixed credentials and stable markup.
Step 1: Browser harness
The harness wraps Playwright and exposes primitive actions. It never interprets the page; it only forwards pixels and mouse commands.
from playwright.sync_api import sync_playwright
import base64
class BrowserEnv:
def __init__(self, width=1024, height=768):
self.width = width
self.height = height
self.playwright = sync_playwright().start()
self.browser = self.playwright.chromium.launch(headless=True)
self.page = self.browser.new_page(viewport={"width": width, "height": height})
# Kill animations that break coordinate stability
self.page.add_style_tag(
content="* { transition: none !important; animation: none !important; }"
)
def goto(self, url):
self.page.goto(url, wait_until="networkidle")
def screenshot(self) -> str:
png = self.page.screenshot()
return base64.b64encode(png).decode("utf-8")
def click(self, x, y):
self.page.mouse.click(x, y)
def type(self, text):
self.page.keyboard.type(text, delay=10)
def key(self, k):
self.page.keyboard.press(k)
def scroll(self, x, y, dx, dy):
self.page.mouse.move(x, y)
self.page.mouse.wheel(dx, dy)
def close(self):
self.browser.close()
self.playwright.stop()Step 2: The computer-use loop
Anthropic’s tool schema is versioned. We pin computer_20241022 and pass the current screenshot as an image block in the user turn. The model returns a tool_use block; we execute it and acknowledge with a tool_result.
from anthropic import Anthropic
client = Anthropic()
TOOL = {
"type": "computer_20241022",
"name": "computer",
"display_width_px": 1024,
"display_height_px": 768,
}
SYSTEM = """You control a browser at 1024x768.
Complete the user's shopping task on the current site only.
When the cart is populated and you reach the checkout overview page, emit action 'finish'.
Never use real payment data; if a card field appears, type 4242424242424242.
"""
def run_agent(env: BrowserEnv, task: str, max_steps=25):
messages = [{"role": "user", "content": task}]
for step in range(max_steps):
shot = env.screenshot()
messages.append({
"role": "user",
"content": [{
"type": "image",
"source": {"type": "base64", "media_type": "image/png", "data": shot}
}]
})
resp = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
system=SYSTEM,
tools=[TOOL],
messages=messages,
)
tool_use = next(b for b in resp.content if b.type == "tool_use")
action = tool_use.input
print(f"step {step}: {action.get('action')} {action.get('coordinate', '')}")
if action["action"] == "finish":
break
execute(env, action)
messages.append({"role": "assistant", "content": resp.content})
messages.append({
"role": "user",
"content": [{
"type": "tool_result",
"tool_use_id": tool_use.id,
"content": "executed"
}]
})Expected checkpoint output after env.goto and first loop iteration:
step 0: screenshot
step 1: type
step 2: click
step 3: screenshotThe model typically requests a screenshot first to orient, then starts interacting.
Step 3: Action execution
The API emits coordinates as [x, y] in CSS pixels relative to the viewport. Map them directly to Playwright’s mouse.
def execute(env: BrowserEnv, action: dict):
a = action["action"]
if a == "click":
env.click(*action["coordinate"])
elif a == "double_click":
env.page.mouse.dblclick(*action["coordinate"])
elif a == "type":
env.type(action["text"])
elif a == "key":
env.key(action["key"])
elif a == "scroll":
env.scroll(action["coordinate"][0], action["coordinate"][1],
action["delta"][0], action["delta"][1])
elif a == "wait":
env.page.wait_for_timeout(800)A sample tool_use payload looks like this:
{
"type": "tool_use",
"id": "cu_01ABC",
"name": "computer",
"input": {
"action": "click",
"coordinate": [488, 322]
}
}If you route through a gateway such as n4n.ai, the same OpenAI-compatible chat endpoint forwards provider cache-control hints and falls back automatically when Anthropic is degraded, but the tool schema and execution code stay identical.
Step 4: Prompting for shopping tasks
Vague instructions produce wandering agents. Embed credentials and a clear stop condition in the task string.
TASK = (
"Log in with username 'standard_user' and password 'secret_sauce'. "
"Add the 'Sauce Labs Backpack' to the cart. "
"Open the cart, proceed to checkout, fill first name 'Test', last name 'User', "
"zip '12345', continue, then finish at the overview page."
)The system prompt already forbids real card entry. We rely on the demo site’s lack of a real payment step.
Step 5: End-to-end run
if __name__ == "__main__":
env = BrowserEnv()
env.goto("https://www.saucedemo.com")
try:
run_agent(env, TASK, max_steps=30)
finally:
env.close()Successful run prints a trajectory ending in finish:
step 0: screenshot
step 1: type
step 2: click
step 3: screenshot
step 4: click
step 5: type
step 6: click
step 7: screenshot
step 8: click
step 9: finishAt that point the browser is at the checkout overview URL. No network call to a payment processor occurred.
Hardening the agent
Computer-use APIs are sensitive to layout drift. Concrete fixes we ship:
- Disable CSS transitions on load (done in
BrowserEnv). - Cap steps and restart from the cart URL if no
finishwithin budget. - Run inside a container with no mounted secrets; the agent only sees pixels.
- Log every
(action, coordinate)tuple to replay failures offline.
import json, time
def logged_execute(env, action):
with open("trace.jsonl", "a") as f:
f.write(json.dumps({"t": time.time(), "a": action}) + "\n")
execute(env, action)Swap execute for logged_execute in the loop to get a forensic trace.
Safety boundaries
A computer use shopping agent can click anything rendered. Sandbox it strictly: no real auth cookies, no saved cards, network egress limited to the target domain. Use test card numbers only. If the model ever types outside a known input region, abort the loop.
The pattern here is model-agnostic. If you later swap Claude for another vision model that exposes a compatible tool interface, the BrowserEnv and execute layers require zero changes.
Where to take it next
You now have a minimal but real computer use shopping agent: a Playwright harness, a versioned computer-tool loop, and a constrained prompt. Production extensions include a DOM-diff fallback when the model stalls, a confirmation gate before finish, and per-token usage metering if you front the calls with an inference gateway. Build the trace log first—debugging pixel agents without it is guessing.