A claude computer use browser agent lets you drive a real Chromium instance through screenshot-and-action loops mediated by Claude 3.5 Sonnet. This tutorial builds a minimal but functional agent using the Anthropic Messages API beta and Playwright—no agent framework required, just a tight observation-execution loop you fully control.
Prerequisites
- Python 3.10+ with
pip - An Anthropic API key (
ANTHROPIC_API_KEYin env) anthropicandplaywrightPython packages- Chromium binary via Playwright
pip install anthropic playwright
playwright install chromium
export ANTHROPIC_API_KEY=sk-ant-...You should be comfortable reading Playwright’s page API and Anthropic’s messages.create signature. The agent we build will navigate to a site and perform a single instructed task, then exit.
The control loop
The core of any claude computer use browser agent is the observation-action loop:
- Capture a screenshot from the browser.
- Send it to Claude with the
computertool enabled. - Receive a
tool_useblock describing a primitive action (click, type, scroll, screenshot, navigate). - Execute that action in Playwright.
- Return the resulting screenshot as a
tool_resultand repeat.
Claude does not see the DOM. It reasons purely from pixels and your system prompt. That constraint shapes everything below.
Tool and beta headers
Computer use is a beta feature. You must pass the anthropic-beta: computer-use-2024-10-22 header and declare the tool with type computer_20241022.
TOOL = {
"type": "computer_20241022",
"name": "computer",
"display_width": 1024,
"display_height": 768,
}
BETA_HEADER = {"anthropic-beta": "computer-use-2024-10-22"}Step 1: Capture and send a screenshot
Start a Playwright page, load a URL, and grab a PNG. Encode it as base64 and send it as an image content block.
import base64
from playwright.sync_api import sync_playwright
from anthropic import Anthropic
client = Anthropic()
def screenshot_b64(page):
png = page.screenshot()
return base64.b64encode(png).decode("utf-8")
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page(viewport={"width": 1024, "height": 768})
page.goto("https://www.google.com")
img = screenshot_b64(page)
resp = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
system="You are a precise browser agent. Use the computer tool to complete the user's task.",
tools=[TOOL],
tool_choice={"type": "tool", "name": "computer"},
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "Search for 'openrouter models'"},
{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": img}}
]
}],
extra_headers=BETA_HEADER,
)
print(resp)Expected first response
Claude will return a tool_use asking to click the search box or type into it. A typical block:
{
"type": "tool_use",
"id": "toolu_01A",
"name": "computer",
"input": {
"action": "click",
"coordinate": [512, 48]
}
}If you get a stop_reason of end_turn instead, your system prompt or task wording is too ambiguous—tighten it.
Step 2: Parse Claude’s tool use
The response content is a list. Find the tool_use block and switch on input.action.
def extract_action(resp):
for block in resp.content:
if block.type == "tool_use":
return block.id, block.input
return None, NoneActions we handle: screenshot, click, double_click, type, keypress, scroll, move, navigate. The beta spec also includes drag, but we skip it for brevity.
Step 3: Execute actions with Playwright
Map each action to a Playwright call. Coordinates from Claude are absolute pixels in the 1024×768 space; Playwright’s viewport matches that.
def execute(action, page):
a = action["action"]
if a == "click":
x, y = action["coordinate"]
page.mouse.click(x, y)
elif a == "double_click":
x, y = action["coordinate"]
page.mouse.dblclick(x, y)
elif a == "type":
page.keyboard.type(action["text"])
elif a == "keypress":
for key in action["keys"]:
page.keyboard.press(key)
elif a == "scroll":
x, y = action.get("coordinate", [0,0])
page.mouse.move(x, y)
page.mouse.wheel(action.get("delta_x", 0), action.get("delta_y", 0))
elif a == "navigate":
page.goto(action["url"])
elif a == "screenshot":
pass # we always re-screenshot below
# move is implicit in click; ignore for minimal agentAfter execution, take a fresh screenshot to feed back.
Step 4: Putting it together
The full loop runs for a fixed number of steps or until Claude returns end_turn.
def run_agent(task, max_steps=10):
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page(viewport={"width": 1024, "height": 768})
page.goto("https://www.google.com")
messages = [{"role": "user", "content": [{"type": "text", "text": task}]}]
for step in range(max_steps):
img = screenshot_b64(page)
# attach latest screenshot to last user message
messages[-1]["content"].append({
"type": "image",
"source": {"type": "base64", "media_type": "image/png", "data": img}
})
resp = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
system="You are a precise browser agent. Complete the task using the computer tool.",
tools=[TOOL],
tool_choice={"type": "tool", "name": "computer"},
messages=messages,
extra_headers=BETA_HEADER,
)
tool_id, action = extract_action(resp)
if not tool_id:
print("Agent finished:", resp.stop_reason)
break
execute(action, page)
# send tool_result with new screenshot
messages.append({
"role": "user",
"content": [{
"type": "tool_result",
"tool_use_id": tool_id,
"content": [{
"type": "image",
"source": {"type": "base64", "media_type": "image/png", "data": screenshot_b64(page)}
}]
}]
})
browser.close()
run_agent("Search for 'openrouter models' and open the first result")Note: we append images to the same user turn before each call. In practice you should alternate user/assistant/tool_result roles strictly; the snippet above collapses for readability. Production code should track roles correctly.
Key checkpoints and output
After the first execute (a click on the search box), the next response typically types:
{
"input": {
"action": "type",
"text": "openrouter models"
}
}Then a keypress with ["Enter"]. The loop continues until the SERP loads and Claude emits end_turn because the task is satisfied. You can assert success by checking page.url or scraping the title.
Limitations and hardening
A claude computer use browser agent is brittle without guardrails:
- Coordinate drift: if the viewport resizes, all cached coordinates break. Lock the viewport.
- Latency: each step is a round-trip plus screenshot encode. Budget ~1–2s per action.
- Safety: Claude can trigger downloads or navigate off-domain. Sandbox the browser and intercept
page.on("popup"). - Cost: every screenshot is sent as a full image. Use
max_tokenstightly and cap steps.
For production traffic, route the API call through a gateway that provides automatic fallback and per-token metering so a single provider outage doesn’t stall your agent. The agent logic stays identical; only the base_url and auth header change.
If you need to scale beyond a single tab, parallelize isolated browser contexts per task and reuse the same loop. The computer use protocol is stateless per call—all state lives in the screenshot and your message history.
Wrap-up
You now have a runnable claude computer use browser agent in ~80 lines of Python. It reads pixels, emits primitive UI actions, and iterates. Extend it with DOM-aware fallback (Playwright selectors when Claude gets stuck) or a planner model that decomposes tasks before the action loop starts. The beta is stable enough for internal tools today; just keep the viewport fixed and the step limit low.