τ-bench: reliable agentic actions need versioned tests and approval gates

External-source research synthesis, not an OfferSignal client result. Based on the retained October 2026 research and source-specific dates.

A service agent can retrieve the right product yet make the wrong state change. τ-bench evaluates an agent that must converse, inspect data, follow policy and call tools—not merely answer a question. It is a laboratory benchmark, not a live customer-service deployment, payment product or procurement case.

What τ-bench actually tested

The original June 2024 study placed an agent in synthetic retail and airline conversations with a language-model-simulated user. The agent could send messages or call APIs that changed a hidden database: returns, exchanges, cancellations, bookings and refunds. Retail used 500 users, 50 products and 1,000 orders; airline used 500 users, 300 flights and 2,000 reservations. The original release reported 115 retail tasks and 50 airline tasks. Tasks allowed up to 30 actions, with deterministic transitions and a mix of API-enforced and prompt-only controls. 1

Success matched the final database state to an annotated goal and checked required response information. The paper’s pass^k estimates the chance that all of k independent replays succeed, rather than only one replay succeeding. The scorer was necessary but not sufficient: an expected end state did not prove explicit confirmation or every interaction rule. 1

The headline result is not dependable service

For GPT-4o function calling, the original paper reports pass^1 of 61.2% in retail and 35.2% in airline, with a 48.2% domain-weighted average rather than a task-count-weighted average. In retail, the best GPT-4o result had pass^8 below 25% despite pass^1 above 60%. These are model- and version-specific results with synthetic users and data, not a real customer-service error rate, human-agent comparison or payment-loss estimate. 1

The paper reviewed 115 retail trajectories: 40 failed, four were repaired as user typos or ambiguities, and the remaining 36 were attributed to agent issues. Wrong tool arguments, incomplete policy following and partial completion of compound requests were examples; compound-request problems represented 19% of those 36 agent failures. A quote or fit enquiry often contains several dependent facts and actions, so a fluent response or correct first lookup does not establish safe completion. 1

Benchmark maintenance changes the comparison

Sierra’s February 2026 τ³ report says it fixed “50+” tasks across wrong expected actions, ambiguous instructions, impossible constraints, missing fallbacks, loopholes, data errors and tool gaps. Its displayed counters show 27 airline and 26 retail fixes. After rerunning three 2026 models, it reports airline pass^1 increases of 14.0–20.0 percentage points and retail changes of -0.4 to +5.5 points; GPT-5.2 airline pass^4 is shown as 50.0% before versus 72.0% after. These are vendor-reported synthetic reruns and benchmark corrections, not model improvement or a like-for-like 2024 GPT-4o comparison. 4

There is an unresolved source-control conflict: the paper reports 115 retail tasks, while the audit describes 114; the audit’s “50+” headline, 27-plus-26 counters and “50 of 50 fixes” footer also do not align. Preserve the task version and raw diff before comparing scores. 1 4

A practical test for product and quote workflows

The independent SABER follow-up audited the tasks into τ-Bench Verified. On its revised Airline set, a Qwen3-Thinking-235B system rose from 58.5% to 78.2% in a three-run average using mutation-gated verification, targeted reflection and context cleaning. It used a Claude Sonnet 4 simulated user and 30-turn cap. Its trace analysis associated an additional mutating-action deviation with a 57–82% lower odds of task success, compared with 7–15% for a non-mutating deviation. This is a benchmark association, not a field causal estimate, and does not show that confirmation overhead works for real customers. 5

For an illustrative equipment or distributor workflow, create versioned fixtures for product identity, fit history, price policy, customer authority, tool schema and permitted state changes. Separate reads from quote, order, refund, supplier-contact and payment writes. Replay ambiguous and compound requests; measure both pass^1 and all-runs-success; retain the trajectory and policy/data versions. The product or commercial owner—not the agent—must approve binding prices, orders or payments. A real-user pilot with a baseline is still required for adoption, accuracy, latency, customer burden or return-on-investment claims.

Sources and next reading

τ-bench official site and leaderboard — Public evaluation resource.

Original τ-bench code and data — Public materials; original tasks are marked outdated.

Current τ³-bench repository — Public current code; scores are not identical to 2024.

Explore OfferSignal · Focused Business & AI Workshop · All research

References