WebArena vs WebVoyager: two browsing agent benchmarks

WebArena vs WebVoyager is the comparison most teams face when they need a reproducible way to score a web browsing agent. Both target the same high-level goal—measuring an LLM’s ability to click, type, and retrieve across websites—but they make opposite tradeoffs on environment control, cost, and realism.

What each benchmark actually measures

WebArena

WebArena ships a self-hosted cluster of web apps (Shopping, Reddit, GitLab, Wikipedia) inside Docker containers. Tasks are fixed scripts: “create a GitLab issue with this title” or “find the cheapest laptop under $500”. The agent interacts through a trimmed DOM observation and action space (click, type, scroll). Success is checked by deterministic post-condition scripts that inspect the app’s database or DOM state.

WebVoyager

WebVoyager drives a real Chromium instance against 15 live websites (Amazon, BBC, Coursera, etc.). It defines 300 open-ended tasks like “book a hotel in Paris for under $200”. The agent receives screenshots plus DOM text. A multimodal judge (GPT-4V) grades the trajectory against a rubric, producing a binary or partial score.

Capabilities: task scope and observation modality

The WebArena vs WebVoyager split is stark: one freezes the world, the other floods it with reality. WebArena’s four sites are synthetic but functionally complete. Its action space is narrow, which keeps agents from going off the rails, but it cannot test handling of cookie banners, layout shifts, or CAPTCHAs. Observation is DOM-only unless you extend it.

WebVoyager embraces the mess of the real web. Screenshots force the agent to deal with visual context, and the 15 sites cover travel, shopping, and content. However, tasks are fewer and the judge introduces variance: two runs of the same trajectory can score differently.

If your agent is multimodal, WebVoyager is the only one of the two that exercises vision. If your agent is text/DOM-only, WebArena’s tighter feedback loop is more informative per compute hour.

Cost model: infrastructure vs API spend

In the WebArena vs WebVoyager cost discussion, the difference is who pays for tokens. WebArena is open-source. You pay for the box that runs the Docker stack—typically a 16GB RAM instance for the four apps. No per-task LLM cost is imposed by the benchmark itself, though your agent will call a model.

WebVoyager imposes direct API costs. Each task step sends a screenshot to a vision model, and the final judge call consumes ~2–4K tokens of image-encoded context. At 300 tasks with ~10 steps each, the judge and agent spend adds up quickly. A single run against GPT-4V-class models can cost hundreds of dollars in API fees alone.

When swapping models, an OpenAI-compatible gateway such as n4n.ai simplifies access: one endpoint covers 240+ models with automatic fallback when a provider is rate-limited, so you don’t rewrite client code to benchmark a new vision model.

Latency and throughput

WebArena tasks run locally. With the containers up, you can parallelize 20+ agent threads on one machine because the environments are isolated and scripted. Typical task wall-time is bounded by your agent’s step latency, not the env.

WebVoyager hits live sites. You inherit their rate limits, CDN throttling, and occasional 503s. Parallelism is limited to what the sites tolerate; aggressive concurrency gets you blocked. A full sweep often runs overnight, not minutes.

Ergonomics: setup and harness

The WebArena vs WebVoyager setup experience mirrors their core assumptions. WebArena provides a docker-compose.yml and a pytest harness:

git clone https://github.com/web-arena-x/webarena
cd webarena
docker compose up -d
pytest tests/test_task.py --task-id 1

The config file maps each site URL and database credential. You implement a single Agent interface that returns actions given observations.

WebVoyager’s repo is less turnkey. You need Playwright, a vision model key, and a judge prompt. A minimal agent loop looks like:

from openai import OpenAI
client = OpenAI()

def step(screenshot_b64, dom_text):
    resp = client.chat.completions.create(
        model="gpt-4-vision-preview",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": dom_text},
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{screenshot_b64}"}}
            ]
        }]
    )
    return resp.choices[0].message.content

Then you run the evaluator, which calls the judge. There is no single command that stands up the world.

Ecosystem and extensibility

WebArena has spawned VisualWebArena (adds screenshots) and is referenced in dozens of agent papers. Its fixed sites make it easy to add tasks: write a Python function that checks state.

WebVoyager is newer. Its contribution is the live-site methodology and judge prompt. Extending it means writing new task specs and hoping the site hasn’t changed its DOM. The community has forked it for specific verticals (e.g., airline booking), but reproducibility decays as sites evolve.

Hard limits

WebArena’s worlds are frozen. Any behavior that depends on live data—stock prices, news—is out of scope. Its Reddit and GitLab are mocked, so auth edge cases are simplified.

WebVoyager’s live dependency is its Achilles heel. A site redesign breaks tasks silently. The LLM judge biases toward verbose or confident trajectories. And you cannot audit why a score dropped without reading the judge’s rationale.

Head-to-head summary

Which to choose

Choose WebArena if you are iterating on a DOM-based agent weekly and need a CI gate. The deterministic checks catch regressions without burning API budget. It fits teams building internal tools where the web is a known set of apps.

Choose WebVoyager if your agent ships to consumers and must handle real layouts, images, and popups. The vision requirement and live sites expose failure modes WebArena hides. Budget for API cost and expect to re-run tasks after site changes.

Choose both if you have resources: WebArena for fast inner-loop dev, WebVoyager for periodic real-world spot checks. The combined signal covers both correctness in a controlled world and robustness in the wild.

For model comparison specifically, run WebArena with a cheap local model to filter candidates, then promote finalists to WebVoyager with a vision-capable endpoint. That sequencing keeps spend sane.