What is computer use AI? It’s a model capability that lets an LLM drive a graphical user interface the way a human does—reading screenshots, moving the mouse, clicking, and typing. Claude’s screen-control API exposes this through a structured tool loop where the model emits low-level input actions and your code executes them against a real or virtual desktop.
How the Claude computer-use loop works
The core is a tight observe–act–observe cycle. You send a screenshot (or let the model request one), the model returns a JSON action, you apply it to the OS, capture a new screenshot, and feed it back. There is no hidden browser automation; the model only sees pixels and emits coordinates.
Tool definition
You declare a computer_20241022 tool in the Messages API. The model then knows the display dimensions and can reason about x/y coordinates.
{
"type": "computer_20241022",
"name": "computer",
"display_width": 1920,
"display_height": 1080
}If you omit the tool, Claude will never attempt screen control. The tool is permission, not a background daemon.
Action types
The model can emit screenshot, mouse_move, left_click, type, key, scroll, and a few others. Each comes back as a tool_use content block:
{
"type": "tool_use",
"id": "toolu_01",
"name": "computer",
"input": {
"action": "click",
"coordinate": [842, 315]
}
}Your job is to map that to X11/Quartz/Win32 calls, then return a fresh screenshot as a tool_result containing an image block. The loop repeats until the model sends a normal text response or hits a stop condition.
Minimal execution loop
import anthropic, base64
client = anthropic.Anthropic()
tools = [{"type": "computer_20241022", "name": "computer",
"display_width": 1920, "display_height": 1080}]
messages = [{"role": "user", "content": "Export last month's CSV from the dashboard"}]
while True:
r = client.messages.create(model="claude-3-5-sonnet-20241022",
messages=messages, tools=tools, max_tokens=1024)
tu = [b for b in r.content if b.type == "tool_use"]
if not tu:
print(r.content[0].text); break
for block in tu:
img = run_action(block.input) # your OS hook returns PNG bytes
messages.append({"role": "user", "content": [
{"type": "tool_result", "tool_use_id": block.id,
"content": [{"type": "image", "source": {"type": "base64",
"media_type": "image/png",
"data": base64.b64encode(img).decode()}}]}
]})The model never touches the filesystem or network directly. It delegates all side effects to your harness.
Coordinate systems and scaling pitfalls
Coordinates are expressed in the pixel space you declared in the tool definition. The model, however, observes a downsampled version of the screenshot—Anthropic feeds it a lower-resolution grid to keep token counts sane. If you declare 1920×1080 but the model sees a 1024-long-edge render, a click at (842,315) in its mental map must be scaled back to native resolution before you call XWarpPointer.
A robust harness maps both ways:
def to_native(coord, declared, observed):
sx = declared[0] / observed[0]
sy = declared[1] / observed[1]
return [int(coord[0] * sx), int(coord[1] * sy)]Multi-monitor setups are not supported; only the primary display is addressed. If your app opens on a second screen, the agent goes blind.
Why screen-control changes agent design
Understanding what is computer use ai helps you decide whether to invest in UI-level automation versus API integrations. Traditional agents call structured endpoints. Computer use throws that out: any software with a GUI becomes callable, including legacy tools, Citrix sessions, and internal apps with no API.
The trade-off is latency and fragility. A single task might require 20–40 round trips, each involving a screenshot encode/decode and a model inference. At 500–1000 ms per step, a simple workflow takes minutes. That is acceptable for back-office RPA but lethal for interactive chat.
Concrete example: pulling a report from a legacy portal
Say you need data from a Java applet that predates REST. You prompt: “Log in, navigate to Reports > Monthly, and click Export.” The model:
- Requests a screenshot.
- Sees the login field at (400, 300), types credentials.
- Clicks Submit at (700, 420).
- Waits for render, screenshots again.
- Identifies the menu, clicks, then Export.
Your harness runs on a headless Linux box with Xvfb:
Xvfb :99 -screen 0 1920x1080x24 &
export DISPLAY=:99
python agent.pyNo API credentials for the legacy system are needed—only a valid user session. That is the real power: you automate what you cannot refactor.
Common misconceptions
The hype cycle around what is computer use ai often ignores operational reality.
“It sees like a human”
Claude processes the screenshot at reduced resolution and tokenizes patches. Fine text in a 1920×1080 image may be illegible. You must scale the display or zoom the app, not assume human acuity.
“It’s deterministic”
Coordinates drift when the window moves or the OS theme changes. A button at (842,315) in one session may be at (838,320) after a notification pops. You need anchoring logic or let the model re-screenshot frequently.
“You can point it at prod”
Giving a model raw mouse control in a production environment is how you get deleted rows. Run it in a disposable VM, snapshot before, and diff after. The model is not aware of consequences beyond pixels.
Security model
Because the model receives screenshots as base64 image blocks, everything visible on screen leaves your network and goes to the inference provider. A banking portal or PII-laden CRM is exposed in plaintext pixels. Use scoped accounts, redact via window masking, or run the agent against a clean virtual desktop with no other apps open.
The model also has no concept of “read-only.” A mispredicted key action sending Ctrl+Shift+Delete can wipe a field. Your harness should intercept destructive key combos and require a human gate for actions outside a allowlist.
Comparison to browser agents
DOM-aware browser agents (Playwright-backed) know element IDs and text nodes. They are faster and cheaper because they skip pixel reasoning. Computer use is universal—it works on native desktop apps, Flash, and remote desktops—but pays a tax in tokens and fragility. Pick browser agents when the target is web; reach for screen control when there is no DOM to parse.
Production routing and resilience
If you route these calls through an OpenRouter-class gateway such as n4n.ai, you get automatic fallback when Anthropic is rate-limited and per-token metering, but the computer-use loop itself is unchanged—you still execute actions and return screenshots. The gateway simply forwards your computer_20241022 tool definition and provider cache-control hints; it does not simulate the desktop.
Cache control matters: mark the system prompt and initial screenshot as ephemeral or long-lived to cut cost on the 30th step. Without it, every iteration re-sends the full image history. At published Sonnet rates, a 1080p screenshot is a fraction of a cent to input, but multiplied by 40 steps and hundreds of daily tasks, the bill grows. Use cache_control on the first screenshot block and reuse it.
Cost and latency math
Assume a task needs 25 steps. Each step: one outbound screenshot (~1.2k input tokens equivalent for image) and a small action output (~50 tokens). At public Claude 3.5 Sonnet pricing, image input is the dominant cost. If you cache the first screenshot and only send deltas, you trim 80% of redundant tokens. Latency-wise, 25 × 800 ms = 20 seconds of pure inference, plus your action execution. Plan for minutes per complex task, not seconds.
Where to start
Stand up a virtual display, wire the tool loop, and pick a task with a clear visual end state. Avoid open-ended goals like “fix the config.” Computer use excels at repetitive GUI paths, not judgment calls. What is computer use ai? A pragmatic bridge for software that lacks an API—not a replacement for one.