Online-Mind2Web: a browser benchmark is not a procurement system

External-source research synthesis, not an OfferSignal client result. Based on the retained October 2026 research and source-specific dates.

A browser agent can follow instructions on a live website and still fail the business question behind the task: applying the wrong price or date filter, confusing a product with an accessory, or explaining an incorrect action convincingly. Online-Mind2Web tests that navigation problem directly—but it is not a shopping, request-for-quote, negotiation, payment or product-data service.

What the benchmark tests

The Ohio State University NLP Group and collaborators assembled 300 authored tasks across 136 websites in shopping, food, housing, finance, travel, government and other domains. Each combines a natural-language instruction with a designated website and start URL. Agents browse, filter, sort, research and sometimes create memes. Starting on the designated site while banning Google Search makes comparison more controlled, but does not represent open-web merchant selection or procurement. 1

This is curated research infrastructure, not a customer-journey sample. In a sample of 650 original Mind2Web tasks, 47% were invalid or stale. The final 300 contain 167 selected original tasks, 24 rewrites, 34 Mind2Web-Live tasks and 75 new tasks; 83 are easy at five or fewer human reference steps, 143 medium at six to ten and 74 hard at eleven or more. Country-level counts are not published, so Nordic or Baltic readiness cannot be established. 1

Read the scores with their protocol

Human-labeled completion rates were 30.7% for SeeAct, 28.0% for Agent-E, 30.0% for Browser Use, 29.0% for Claude Computer Use 3.5, 56.3% for Claude 3.7 and 61.3% for Operator. Each used 300 tasks with two or more annotators and a resolver, but environments differed: open-source agents had 25-step caps, Claude 3.7 had 50 steps with thinking disabled, and Operator used a remote browser. These are benchmark outcomes, not commercial results. 1

The paper attributes 57.7% of Operator errors to filtering or sorting and 19.6% to navigation, including wrong numerical and temporal filters. That is not a market error rate. It is a practical case for representing price, fit, unit and date as checkable fields, retaining the requested constraint, applied filter, observed result and source provenance.

Online-Mind2Web v2 records action, thought, screenshot and URL per step. WebJudge derives task key points, selects screenshots and makes a binary judgment from evidence and action history; official guidance rejects final agent prose because it may hallucinate. Against human labels, o4-mini WebJudge reached 85.7% agreement with a 3.8-point average success-rate gap, while WebJudge-7B reached 87.0% and a 3.9-point gap. This supports iteration, not ground truth. 1 2

Maintenance is part of measurement: the repository records 36 invalid or CAPTCHA tasks replaced on 3 November 2025—12% of 300—with later updates through 15 May 2026. Report task version, run date, unavailability policy, model or harness and evaluator beside every score. 2 3

Browser Use reported 291/300, or 97%, and separately 291/298, or 97.7%, after excluding two tasks it called impossible. This is a vendor self-evaluation scored by its Claude final-response judge, using a different evidence standard from official v2—not demonstrated official human evaluation or customer performance. Careerflow describes more-than-one reviewer, quality assurance and final review, but publishes no denominator, agreement, adjudication policy or validated score. 2 4 5 6

An illustrative business test

For an illustrative distributor, apply the benchmark pattern to one product family before granting an agent access to supplier messages or carts. Create versioned tasks to identify the correct variant, apply unit and fit constraints, compare current prices and dates, and abstain when a fact is missing. Store SKU, source URL, timestamp, action evidence, task version and correction owner. Keep product and quote truth separate from browser navigation; the catalogue, commercial system or named technical owner remains authoritative.

Replay clean, stale, ambiguous and near-duplicate cases. Measure constraint application, wrong-item rate, source-linked accuracy, freshness, clarification and reviewer correction—not just completion. Use automatic judging for iteration, while requiring human approval before supplier messages, cart changes, commitments or payment. Online-Mind2Web is a pre-production realism gate, not a safety, market-readiness or return-on-investment certificate.

Sources and next reading

Official repository — code, task updates, v2 schema and WebJudge guidance. Public browsing; research infrastructure only.

Dataset card — task fields and maintenance policy. Files require logged-in acceptance; English and no regional coverage claim.

Official leaderboard — project-linked application; no placement is asserted.

Explore OfferSignal · Focused Business & AI Workshop · All research

References