The Claude computer use API desktop agent pattern lets a multimodal model drive a real operating system by reading screenshots and emitting mouse and keyboard actions. This guide walks through a working architecture, the exact API contract, and the failure modes you will hit when you move past a demo.
Architecture: the observe-act loop
A desktop agent is a tight loop: capture the screen, send it to Claude, execute the returned action, repeat. The model never touches the OS directly. Your harness translates model output into pyautogui calls and feeds back a fresh screenshot as the observation.
The Claude computer use API desktop agent expects you to manage this loop. The model receives a tool named computer and returns structured tool_use blocks. You execute them and return a tool_result containing the next screenshot.
Capturing and acting on the desktop
Use mss for fast screen capture and pyautogui for input. Keep the display dimensions honest—Claude reasons about pixel coordinates relative to the declared resolution.
import mss
import pyautogui
from PIL import Image
import io
def grab_screenshot():
with mss.mss() as sct:
monitor = sct.monitors[1] # primary display
raw = sct.grab(monitor)
img = Image.frombytes("RGB", raw.size, raw.bgra, "raw", "BGR")
# downscale to reduce payload
img.thumbnail((1280, 800))
buf = io.BytesIO()
img.save(buf, format="JPEG", quality=70)
return buf.getvalue()
def execute_action(action, **kwargs):
if action == "mouse_move":
pyautogui.moveTo(kwargs["coordinate"][0], kwargs["coordinate"][1])
elif action == "left_click":
pyautogui.click()
elif action == "type":
pyautogui.write(kwargs["text"], interval=0.01)
elif action == "key":
pyautogui.press(kwargs["text"])
# screenshot action is handled by the loop itselfCommon pitfall: pyautogui uses the active display’s coordinate space. If you downscale the screenshot before sending, you must map Claude’s coordinates back to full resolution. Multiply returned coordinates by actual_width / sent_width.
The API call shape
Anthropic’s SDK exposes the computer tool via a typed beta tool. The tool name is fixed; you supply display dimensions.
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
tools=[{
"type": "computer_20241022",
"name": "computer",
"display_width_px": 1280,
"display_height_px": 800
}],
messages=[
{"role": "user", "content": "Open the terminal and run ls."}
]
)The first response will typically contain a tool_use block with input like {"action": "mouse_move", "coordinate": [640, 400]}. Your harness executes it, grabs a new screenshot, and sends a tool_result.
# after executing the action from above
screenshot = grab_screenshot()
followup = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
tools=[{"type": "computer_20241022", "name": "computer", "display_width_px": 1280, "display_height_px": 800}],
messages=[
{"role": "user", "content": "Open the terminal and run ls."},
{"role": "assistant", "content": response.content},
{"role": "user", "content": [
{"type": "tool_result", "tool_use_id": tool_use.id, "content": [
{"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": screenshot.decode("latin1")}}
]}
]}
]
)Note: the tool_result must carry the new observation as an image. The model conditions its next action on that image, not on a text description.
Action types and edge cases
The computer tool defines a fixed set of actions. The ones you will implement:
screenshot— return a fresh frame (your loop does this automatically).mouse_move— move without clicking; useful for hover menus.left_click,right_click,double_click— clicks at current or specified coordinate.type— send a string; respects keyboard layout.key— press a single key likeReturnorEscape.scroll— vertical scroll by a delta.
A frequent bug: sending type with a newline character. pyautogui.write does not map \n to Enter. Translate \n to key: "Return" or use pyautogui.press("enter").
Coordinate mapping
If you declare display_width_px: 1280 but capture at 2560 physical pixels, multiply Claude’s coordinates by 2 before pyautogui.moveTo. Mismatches cause silent mis-clicks that are painful to debug.
Context window and image payloads
Each screenshot is a token cost. At 1280×800 JPEG quality 70, you ship roughly 30–60 KB base64, which the model embeds as visual tokens. Over a 20-step task, that adds up.
Tradeoff: lower resolution speeds inference and cuts cost but degrades the model’s ability to read small text. For most desktop UI, 1280×800 is a floor. If you need fine OCR, crop the region of interest and send a zoomed image as a separate user message rather than shrinking the whole screen.
Keep the message history trimmed. You do not need to resend every prior screenshot; keep the last 2–3 observations and the full action sequence as text. Claude tracks state from the actions, not from replaying all images.
Error handling and provider reliability
The loop will break on rate limits, network stalls, or malformed tool outputs. Wrap the API call in a retry with backoff. If a tool_use arrives with an unknown action, skip and send a tool_result containing an error text image (e.g., a red rectangle) so the model can recover.
If you scale this out, an inference gateway such as n4n.ai can provide automatic fallback when a provider is rate-limited, keeping your Claude computer use API desktop agent loop alive without custom retry logic. Its per-token metering also helps you attribute cost per agent session.
Streaming is not your friend here
Tool use is returned only after the full response generates. Streaming the text helps for chat, but for desktop control you need the complete tool_use block before acting. Do not pipe partial deltas to pyautogui—you will get half-formed coordinates.
Security and sandboxing
A Claude computer use API desktop agent has full control of the machine it runs on. Never run it on a host with credentials in the clipboard or an unlocked password manager. Use a throwaway VM or a container with no mounted secrets.
Set pyautogui.FAILSAFE = True so a rapid mouse corner move aborts the script. Add a human interrupt: watch for a specific pixel pattern or a hotkey that pauses the loop. In production, we gate every type action behind a length check to block prompt-injection from a webpage the agent visited.
Latency and throughput tradeoffs
Each step costs one round-trip plus screenshot capture (typically 50–150 ms) and model inference (1–4 s for Sonnet). Parallelizing observation and action is impossible because the next action depends on the new frame.
If you need higher throughput, batch multiple independent desktop sessions across VMs, but keep each session single-threaded. The model’s sequential reasoning does not benefit from concurrent tool calls on the same screen.
A minimal loop skeleton
def run_agent(task_prompt, max_steps=30):
messages = [{"role": "user", "content": task_prompt}]
for _ in range(max_steps):
resp = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
tools=[{"type": "computer_20241022", "name": "computer",
"display_width_px": 1280, "display_height_px": 800}],
messages=messages
)
tool_use = next(b for b in resp.content if b.type == "tool_use")
action = tool_use.input["action"]
if action == "screenshot":
img = grab_screenshot()
else:
execute_action(**tool_use.input)
img = grab_screenshot()
messages.append({"role": "assistant", "content": resp.content})
messages.append({"role": "user", "content": [
{"type": "tool_result", "tool_use_id": tool_use.id, "content": [
{"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": img.decode("latin1")}}
]}
]})
if "task complete" in str(resp.content).lower():
breakThis skeleton omits retries and coordinate scaling but shows the contract. Build from here, add logging of each action to a structured file, and you have a debuggable desktop agent.
Where to invest next
The Claude computer use API desktop agent is viable today for scripted back-office tasks: form filling, report generation, legacy app automation. The hard parts are not the model—they are screenshot compression, coordinate math, and safe execution. Get those right and the multimodal loop becomes a reliable worker.