Pick One API Out of 1,000, Succeed 5% of the Time: A SLATE Benchmark Review

The Paper

Rongzhe Wei, Ge Shi, Min Cheng, Sarthak Ghosh, Vaibhav Gorde, Leman Akoglu, Pan Li, Na Zhang, "SLATE: A Large-Scale Benchmark for Long-Horizon API Selection in E-commerce Tool Use", arXiv:2611.03982 [cs.CL], submitted 2026-11-02. Affiliation: Amazon (7 authors), Georgia Institute of Technology (Pan Li). Presented at the NeurIPS-affiliated workshop "Agentic AI Benchmarks and Applications for Enterprise Tasks." Dataset, simulator, and evaluation harness released (geshijoker.github.io/slate/).

This is a workshop paper that has not gone through full peer review (arXiv listing itself is independent of peer review). Seven of the eight authors are Amazon-affiliated, and the remaining author (Pan Li) is at Georgia Tech, so funding and affiliation concentrate heavily at Amazon. That said, the paper's central finding — that current models fall well short on large-scale enterprise tool selection — is a critical, not promotional, result, and the evaluated models are open third-party systems (Qwen, Gemma, SmolLM, Tulu), not Amazon's own commercial agents, which weakens the case for funder bias.

Study Overview

The research question: when an enterprise with hundreds or thousands of internal APIs hands a standard operating procedure (SOP) to an LLM agent, can the agent select and compose the right tools over a long horizon? The authors argue existing benchmarks fall short on this exact regime — small tool-count settings (τ-bench, AlfWorld) allow controlled evaluation but don't reflect large tool spaces, while benchmarks with larger libraries tend to grade single-step invocation or retrieval, or fall back on subjective LLM-as-a-judge scoring. SLATE pairs a library of roughly 1,000 synthetic e-commerce APIs with 100 tasks built from queries decomposed into hierarchical substeps. Rather than implementing 1,000 live APIs, a Claude-generated "context-aware simulator" returns deterministic outputs for each tool call, and a task is scored as a success only if it reaches every reference outcome — an automatic, reproducible metric with no LLM judge involved.

Key Results

All figures below are read directly from the paper's tables. Task success and step-wise accuracy use different bases and are kept separate — task success requires the entire task to complete correctly, while step-wise accuracy is the correctness rate on a single step.

Source: paper Table 2 and Table 3. The 100 tasks draw on a library of 1,000 tools (417 genuinely relevant, 583 distractors); tasks average 10.8 substeps (range 7-14), and every step picks one of 52 candidates that always includes the correct tool. Step-wise accuracy (36.6-74.8%) looks respectable in isolation, but because an average of 10.6 steps must chain correctly for a task to succeed, errors compound. The paper's own arithmetic shows that a 75%-accurate model chaining 10.6 independent steps would succeed about 4.7% of the time — almost exactly where the best observed task success (5.0%, Gemma-4-12B-it) lands.

Fine-tuning a dedicated "decision model" for selection did not help uniformly. In four of six backbone/decision-model pairs, the decision model beat its backbone's step-wise accuracy by 6.6-15.7 points; but both Strands-family decision models trailed their own Gemma backbones by 17.8 and 20.3 points. Cross-checking against the paper's Table 5 (error-type breakdown) confirms internal consistency — for Gemma-4-12B-it, subtracting its error shares (22.6% wrong tool + 2.1% missed + 0.9% spurious) from 100 gives 74.4%, matching Table 3's step-wise accuracy exactly. The paper's Table 6 (results by task category) showed wide variance, unsurprising given only 6-13 tasks per category: the best task-success rate per category ranged from 3.0% (buyer pre-purchase questions) to 22.2% (gifts & discount codes) — a spread that small samples can easily produce from just one or two wins.

Credibility Assessment

Three things support trust. First, the deterministic simulator plus end-to-end task-success metric removes LLM-judge subjectivity, and every reference solution was pre-verified to pass the simulator. Second, repeating with three different option orderings produced a standard deviation of at most 1.3 points, so results aren't an artifact of option-ordering bias. Third, the reference tool is always guaranteed to be among the 52 candidates, isolating pure selection ability from retrieval failure.

Caveats are equally real. This is a workshop paper without full peer review, and both the queries and the simulated outputs are Claude-generated synthetic data, which may not capture the full irregularity of real enterprise APIs. To avoid rewarding the generator family, the authors excluded Claude models from evaluation — which means the performance of GPT, Claude, and other closed frontier models actually used in production is simply unknown from this paper; the evaluated range is six open 3B-12B model pairs. The authors also report a counterintuitive finding worth flagging: accuracy on substeps whose description names the reference tool verbatim (16.1% of all tool steps) is not higher than on other substeps. That could mean models fail to exploit even explicit hints consistently, or it could mean the dataset's construction makes that hint less useful than it looks — either reading warrants caution.

Related Work

Because this session's egress was blocked, the related papers below were independently confirmed via WebSearch (title, authors, venue, and key figures) rather than by re-reading the primary sources directly. The figures cited here are used only for comparison, not folded into the "Key Results" section above.

• Yao et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — Prior work. An earlier tool-use benchmark covering retail and airline domains, evaluating agent-user dialogue and policy compliance with a small, single-domain tool set. SLATE extends the same "tool use" axis to roughly 1,000 tools across 11 task categories.
• Trivedi et al. (2024, ACL, Best Resource Paper). AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents — Prior work. Implements 9 apps and 457 live APIs, scoring 750 tasks with state-based unit tests; the top model (GPT-4o) solved about 49% of normal tasks. SLATE swaps live APIs for a simulator and more than doubles the tool count (1,000); the drop from 49% to 5.0% underscores how much tool-space and step-count expansion drives up difficulty.
• He et al. (2025, ICLR 2026). TRAJECT-Bench: A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use — Extension/comparator. Shares Amazon authorship with SLATE and scores trajectory-level (sequential/parallel call) accuracy over 1,000+ tools across 10 real-world domains. By SLATE's own Table 1 classification, however, TRAJECT-Bench does not use an end-to-end task-success metric, making it a useful contrast between "is the trajectory right" and "did the task actually complete."

Taken together, tool-use benchmarks have scaled from "dialogue and policy compliance over a small tool space" (τ-bench) to "coding tasks over hundreds of live APIs" (AppWorld) to "trajectory accuracy over thousands of tools" (TRAJECT-Bench). SLATE sits at the intersection of scaling the tool space further while scoring strictly by end-to-end success, and the sharp drop relative to AppWorld quantifies the cost of that tool-space expansion. Nothing here contradicts the prior work.

Reviewer's Take

First, I judge the more important finding is not the 5.0% headline itself but the structure behind it: step-wise accuracy in the 65-75% range collapses to single digits once it compounds across a chain. A single-step "tool-call accuracy: XX%" metric, the kind vendors commonly publish, risks badly overstating the felt success rate of real SOP automation chained across 10+ steps.

Second, I judge that fine-tuning a dedicated decision model should not be assumed to help by default. Both Strands-family pairs lost 17.8-20.3 points versus their own backbone, directly contradicting the industry's implicit assumption that a selection-specialized fine-tune is strictly an upgrade. Any team adopting a tool-selector wrapper should A/B it against the backbone alone before trusting it.

Third, the 3.0-22.2% spread in best-category task success is a signal that the headline number shouldn't be applied uniformly to your own SOPs. The per-category sample (6-13 tasks) is too small to be statistically solid, but the direction — narrow, repetitive SOPs (gift codes, discounts) looking easier than open-ended ones (pre-purchase questions) — is still useful for prioritizing which SOPs to automate first.

Practical Takeaways

• Add step-level checkpoints — for chains of 10+ steps, insert verification points mid-chain so a human can intervene before errors compound to the end.
• A/B any tool-selector before shipping it — don't assume a decision-model wrapper or selection-specialized fine-tune helps; compare it against the backbone alone, since it can regress performance by up to 20.3 points, as the Strands cases show.
• Gauge difficulty by distractor density — a larger share of semantically similar APIs raises selection failure rates, so clearer, more distinguishable internal API naming and descriptions directly improve selection accuracy.
• Prioritize pilots by SOP category, not the overall average — track success rates per category and automate narrow, repetitive SOPs first, keeping open-ended SOPs under human review.
• Separately verify closed frontier models — this paper only covers open 3B-12B models, so the GPT/Claude-class models you'll actually deploy need their own benchmark run.

Conclusion

SLATE puts a number on something practitioners have long sensed: agent success collapses as tool count and chain length grow. Models that hit 65-75% on individual steps still fall to 5.0% once an average of 10.6 steps must chain correctly, which means a single vendor-published metric cannot tell you how an enterprise agent will actually perform in production. Given the pre-peer-review workshop status and the narrow range of open small models tested, read these numbers as a directional signal — frontier closed models might do better — and re-verify against your own SOP and API environment before relying on them operationally. For the operational flip side of this problem — cutting tool exposure rather than measuring selection among many tools — see Cloudflare's Code Mode migration checklist (sunny34.com blog), which tackles the same tension by shrinking 2,594 tools down to 2.

References

• SLATE: A Large-Scale Benchmark for Long-Horizon API Selection in E-commerce Tool Use — arXiv abstract (primary source)
• Full HTML text of the paper — used for table/number verification
• τ-bench (Yao et al., 2024) — related work, primary source
• AppWorld (Trivedi et al., 2024) — related work, primary source
• TRAJECT-Bench (He et al., 2025) — related work, primary source
• Cloudflare's Code Mode migration checklist — sunny34.com blog