WebShop: product selection is not permission to buy

External-source research synthesis, not an OfferSignal client result. Based on the retained October 2026 research and source-specific dates.

A shopping agent can find a plausible product and still be unable to buy it. The WebShop benchmark makes that boundary unusually clear: a policy searches a catalogue, opens item pages, selects attributes and options, and reaches a simulated Buy action. It does not control payment, shipping, merchant acceptance, returns or fulfilment. The result is a useful evaluation environment, not a consumer service or commercial-purchase result.

A realistic catalogue with a controlled task

Princeton natural-language-processing researchers built WebShop from 1,181,436 public Amazon product records across beauty, food, fashion, furniture and electronics, collected from 313 sub-category queries. The dataset contains 842,849 unique option values and 670 manually screened hidden attributes. Search uses an offline deterministic BM25 index, so this is not Amazon’s live ranking or a merchant catalogue feed. The requests were also constructed rather than observed demand: Amazon Mechanical Turk workers wrote 12,087 instructions averaging 15.9 words. The split was 10,587 training, 1,000 development and 500 test requests. Detailed methods identify 1,566 trajectories, while the abstract and project shorthand say more than 1,600; the retained sources do not reconcile that difference. 1 2 3 4

The terminal reward combines product type, hidden attributes, selected options and price. Task Score is 100 times mean reward; success is the share of episodes with reward one. These are constructed task measures, not checkout conversion, customer satisfaction, retail error rate or quoted-total accuracy.

What the reported results actually show

Across three trials on the 500-instruction independent test split, the rule baseline scored 45.6 Task Score and 9.6% success. Imitation learning scored 59.9 and 29.1%; imitation learning plus reinforcement learning scored 62.4 and 28.7%. Selected expert human workers scored 82.1 and 59.6% success, while average-human success was reported as 50%. Thus the familiar 29% versus 9.6% versus 59% comparison is a controlled comparison of policies and selected workers, not a retail benchmark. Imitation learning had the highest exact success, while the combined method had the higher average reward. 1

The failure mechanism is more informative than the headline. Humans exceeded agents by 28 percentage points on option score. Reinforcement learning improved attribute, type and price scores but reduced option score from 45.2 to 38.9 and shortened paths. A choice oracle with hidden reward, attributes and options, plus exhaustive exploration beyond 100 steps, raised rule success from 9.6% to 52.6% and imitation-learning success from 29.1% to 57.6%. This identifies variant resolution and exploration as bottlenecks; the oracle is not commercial authority because it sees hidden product truth. 1

A manual audit of 100 average-worker and 100 expert-worker trajectories compared automatic and manual scores using lexical and synonym allowances. Average workers scored 74.9 automatically versus 76.3 manually, with Pearson correlation 0.856; experts scored 81.5 versus 89.9, with correlation 0.773. Eighty-seven percent of average automatic scores were within 10 points. That supports agreement with the study rubric, not live intent, stock, delivery or willingness to pay. 2

Transfer to real sites still stopped before buying

For 100 sampled instructions, zero-shot imitation learning plus reinforcement learning reached 65.9 score and 25% success on Amazon, versus the rule policy’s 45.8 and 19%. On eBay it reached 62.3 and 21%, versus 31.7 and 7%. The experiment suggests that a navigation pattern transferred across a search and product-domain shift, but it was navigation only: scripts fetched HTML, mapped actions to URLs, and kept option clicks, forms and Buy inside the simulated interface. No order, payment, address, delivery, merchant acceptance or fulfilment occurred. The reported under-eight-second agent versus 815-second human comparison is navigation time, not end-to-end purchase time. 1 2

An illustrative business use

For a manufacturer or distributor, WebShop is best treated as a pre-production regression harness for query reformulation, equipment fit and variant matching—not evidence of autonomous commerce. An illustrative test should use current merchant-provided records, canonical product and variant identifiers, structured compatibility, attributes, price and availability, plus local-language requests. A technical or commercial owner should remain authoritative for substitutions, quoted totals and irreversible order or payment steps.

Measure product match, variant match, quoted-total accuracy and clarification rate separately. Include near-duplicate products, missing options, stale prices and requests that should trigger a question. Then test order acceptance and post-purchase outcomes in a separately governed field study. Do not extrapolate this US-English Amazon simulation to Nordic, Baltic or wider-European commerce without local-language, legal, tax, delivery, assortment and field validation. The public project, source code and listed demo are useful for understanding access and interaction, but availability and behaviour were not tested, and none is evidence of a completed purchase. 3 4 5

Sources and next reading

WebShop official project — public project page and simulated environment.

WebShop source code — benchmark environment, baselines and sim-to-real scripts.

Listed Amazon/eBay query demo — product-finding demo; not a payment or ordering service.

Explore OfferSignal · Focused Business & AI Workshop · All research

References