Sierra's tau2-bench: coordination is harder when authority is split

External-source research synthesis, not an OfferSignal client result. Based on the retained October 2026 research and source-specific dates.

A service agent may know which backend action is needed while the customer controls the device, confirmation or document that makes it possible. Sierra Research's tau2-bench studies that hand-off in a controlled simulation, not a real customer journey.

What the Telecom benchmark tested

The new Telecom domain is a mock technical-support environment. The agent receives CRM-style data and backend actions; a simulated user has a mocked phone and can inspect or change device state. Each side has distinct observations and tools, and only one acts per turn. For example, the agent can inspect a customer record or enable roaming while the user can toggle airplane mode. The paper models this as a two-party partially observed decision process and checks the final state. This is agentic action in a laboratory environment, not a live telecom integration. 1 2

The fixture contains 5 plans, 9 lines and 4 customers. The agent has 7 read and 6 write tools; the language-model user has 15 read and 15 write tools. Fifteen atomic subtask groups produced 2,285 verifiable tasks, from which a balanced 114 were sampled. Telecom success uses final-state assertions, not measures of catalogue quality, customer comprehension, commercial fairness or live failure rates. 1

The central design is authority. In dual-control mode, the agent must explain and coordinate user-side steps rather than silently taking them over. A No-User control gives the agent all tools, while an Oracle-Plan control supplies the required sequence. The comparison separates backend tool competence from communication with a separately authorised actor. 1

Measured results, not customer outcomes

Four API models were tested—GPT-4.1, GPT-4.1-mini, o4-mini and Claude 3.7 Sonnet—with GPT-4.1 acting as the simulated user. Each task ran four times at temperature 0. On Telecom, reported pass^1 was 34% for GPT-4.1, 42% for o4-mini and 49% for Claude 3.7 Sonnet; GPT-4.1-mini is described as around 50%. GPT-4.1's Telecom result contrasts with 74% in Retail and 56% in Airline. These are benchmark successes with a mocked state and model-simulated user, not customer-resolution rates or service KPIs. 1

Moving from No-User to default dual control reduced pass^1 by 18 percentage points for GPT-4.1 and 25 points for o4-mini. Default-mode success was close to zero for tasks requiring more than seven actions. Within this synthetic distribution, coordination and long action sequences added substantial failure beyond tool selection alone. The result cannot be converted into a forecast for a buyer, supplier or service workflow. 1

The simulator also introduced a limitation. Two annotators reviewed 50 Telecom conversations; the table row and prose report 8/50 errors (16%), including 3/50 critical (6%) and 5 benign (10%). The Table 2 caption instead says there were no critical Telecom errors. The defensible reported numerator is 3/50, with the inconsistency disclosed. No inter-annotator-agreement statistic is reported, and an LLM user's behaviour is not human validation. Peer review notes the single truly dual-control domain, absence of enterprise or human-in-the-loop agents, and the deliberately simple vanilla-agent loop. 1 5

Version control matters

Sierra's original v0.1.0 release, tagged 12 June 2025, provides the Telecom, Retail and Airline framework, task and policy materials, and No-User, Oracle-Plan and workflow-policy modes. It is an evaluation artifact, not a deployed service application. 3 4

The 2026 v2.0 release changed 50 English Telecom tasks, prompts and personas, communication scoring, the natural-language judge and lookup behaviour. Its changelog says 2.0 scores are not comparable with 1.x and no flag combination exactly reproduces the 1.0.1 inputs. Comparisons need a pinned task set, prompts, scorer, model versions, trial count and error treatment; the public leaderboard is not a continuous series of comparable customer results. 6 7

An illustrative buyer–supplier test

This is an illustrative application, not a reported deployment. A manufacturer or distributor could let an agent retrieve governed product attributes, draft a compatible option and prepare a quote, while the buyer, technical owner or finance owner retains authority over fit facts, substitutions, exceptions and payment release. Test the same scenario with all tools and with dual control, checking the commercial end state and approval trail.

Keep the product record, quote version, required evidence, actor permissions and final confirmations machine-readable. Measure wrong-variant and unsupported-claim rates, clarification loops, abandoned hand-offs and review time—not just fluent conversation or a successful demo. Human testing should cover unclear instructions, incorrect inputs, consent and exception recovery. The benchmark's useful lesson is an authority map; it is not evidence of savings, conversion, payment reliability or live autonomy.

Sources and next reading

Explore OfferSignal · Focused Business & AI Workshop · All research

References