Benchmarking browser-use agent latency across models

A browser-use agent latency benchmark we ran across five hosted models shows that the LLM, not the browser automation layer, is the dominant factor in end-to-end task time. We standardized a 12-task web workflow suite and swapped only the model endpoint to isolate reasoning and token generation cost. The results reshape how you should pick a brain for production browser agents.

Test setup

We used the open-source browser-use Python package driving Playwright Chromium, pinned to a single us-east worker to remove network jitter. Tasks ranged from “log into a test site and screenshot the dashboard” to “find the cheapest flight and book with dummy data” — each requiring 5–15 agentic steps.

To avoid rewriting code per provider, we routed every call through one OpenAI-compatible endpoint that addresses 240+ models and supports automatic fallback when a provider is degraded. Swapping models was a one-line model= change.

from browser_use import Agent
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="claude-3-5-sonnet",
    base_url="https://api.n4n.ai/v1",
    api_key="YOUR_KEY",
    temperature=0.0,
)
agent = Agent(task="Extract Q3 revenue from sec.gov", llm=llm)
await agent.run()

We measured wall-clock per step, tokens in/out, and task success rate. Pricing uses public list rates as of Q3 2024; your gateway may differ.

Models under test

• GPT-4o (OpenAI): flagship multimodal, 128k context.
• Claude 3.5 Sonnet (Anthropic): strong agentic reasoning, 200k context.
• Gemini 1.5 Pro (Google): million-token context, variable throughput.
• Llama 3.1 70B (hosted open weight): no native vision, 128k context.
• Mixtral 8x22B (hosted): sparse MoE, cheap, weaker instruction following.

Head-to-head comparison

Dimensions explained

Capabilities

Browser agents live or die on instruction following and the ability to recover from unexpected DOM states. In our browser-use agent latency benchmark, Claude 3.5 Sonnet completed 11/12 tasks on first try; GPT-4o did 10/12. Llama 3.1 70B managed 7/12 because it couldn’t interpret screenshots without a separate vision model piped in. Mixtral stalled on pagination.

Price / cost model

List prices matter only when multiplied by token volume. A typical task burned 8k–20k output tokens. At those scales, Gemini’s $5/MTok out is ~3x cheaper than Sonnet’s $15. But if the cheap model fails and retries, the effective cost inverts. We tracked cost-per-successful-task, not raw rate.

Latency / throughput

The browser-use agent latency benchmark confirmed that time-to-first-token (TTFB) drives interactive feel more than raw tokens/sec. Sonnet’s lower TTFB made the agent appear snappier despite similar step counts. Llama’s fast TTFB was offset by extra corrective steps. Gemini showed periodic 1–2s TTFB tails that broke the agent’s action timeout.

{
  "step_latency_ms": {
    "gpt-4o": 1200,
    "claude-3-5-sonnet": 950,
    "gemini-1.5-pro": 1500,
    "llama-3.1-70b": 800,
    "mixtral-8x22b": 700
  },
  "note": "Median wall-clock per agent step, inclusive of browser action"
}

Numbers above are median observations from our run, not vendor specs; treat as relative.

Ergonomics

OpenAI-compatible APIs win for prototyping. browser-use expects a LangChain chat model, and every model here except native Anthropic/Google can be wrapped as ChatOpenAI with a base_url. Sonnet and Gemini require either native SDKs or a translation gateway. If you already use MCP, Sonnet has first-class support.

Ecosystem

GPT-4o has the most prior art: every agent framework has a worked example. Claude’s agentic community is catching up fast with MCP servers. Gemini’s million-token context is unmatched for ingesting entire documentation sites before acting. Llama/Mixtral shine where you need to spin up a private endpoint.

Limits

Context windows are not just size—they’re reliability boundaries. We hit Gemini’s region throttling after 50 consecutive tasks. Sonnet’s abuse filters occasionally refused synthetic login pages. Llama’s lack of vision forces a parallel OCR service, adding latency and a failure mode.

Running your own benchmark

Don’t trust a single vendor’s claims. Stand up the same browser-use task suite and log per-step timing.

export OPENAI_BASE_URL=https://your-gateway/v1
python run_bench.py --model claude-3-5-sonnet --tasks tasks.yaml

Capture step_latency, tokens, success. The browser-use agent latency benchmark you run internally will reflect your own DOM complexity, which matters more than our numbers.

Which to choose

High-reliability interactive agents (customer-facing): Claude 3.5 Sonnet. Lowest median step latency, best recovery, 200k context covers long sessions. Pay the premium per token.

Cost-sensitive batch scraping: Gemini 1.5 Pro when tasks fit its region limits, or Llama 3.1 70B self-hosted if you can add vision OCR. The browser-use agent latency benchmark shows Gemini wins on cost-per-success at scale.

Rapid prototyping with max community support: GPT-4o. You’ll find every snippet, and the latency is acceptable.

Privacy-required on-prem: Llama 3.1 70B via llama.cpp. Accept the extra engineering for vision and the higher retry rate.

Throwaway experiments: Mixtral 8x22B only if you want cheap text-only loops and can hand-hold the agent.

Model choice is the lever. The browser automation layer is commodity; the LLM is the bottleneck. Benchmark it like you mean it.