![[2510.25726] The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2510.25726] The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution
Summary
The Tool Decathlon introduces a benchmark for evaluating language agents on diverse, realistic, and complex tasks, highlighting significant performance gaps in current models.
Why It Matters
This research addresses the limitations of existing benchmarks for language agents, which often focus on narrow tasks. By providing a comprehensive evaluation framework with realistic environments and diverse applications, it aims to enhance the development of more capable agents for real-world applications, thus pushing the boundaries of AI capabilities.
Key Takeaways
- Tool Decathlon benchmarks language agents across 32 applications and 604 tools.
- Existing models show significant shortcomings, with the best achieving only a 38.6% success rate.
- The benchmark includes 108 tasks requiring multi-step interactions, emphasizing real-world applicability.
- Toolathlon aims to drive improvements in long-horizon task execution for language agents.
- Realistic environment states enhance the evaluation process, offering a more comprehensive assessment.
Computer Science > Computation and Language arXiv:2510.25726 (cs) [Submitted on 29 Oct 2025 (v1), last revised 26 Feb 2026 (this version, v2)] Title:The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution Authors:Junlong Li, Wenshuo Zhao, Jian Zhao, Weihao Zeng, Haoze Wu, Xiaochen Wang, Rui Ge, Yuxuan Cao, Yuzhen Huang, Wei Liu, Junteng Liu, Zhaochen Su, Yiyang Guo, Fan Zhou, Lueyang Zhang, Juan Michelini, Xingyao Wang, Xiang Yue, Shuyan Zhou, Graham Neubig, Junxian He View a PDF of the paper titled The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution, by Junlong Li and 20 other authors View PDF Abstract:Real-world language agents must handle complex, multi-step workflows across diverse Apps. For instance, an agent may manage emails by coordinating with calendars and file systems, or monitor a production database to detect anomalies and generate reports following an operating manual. However, existing language agent benchmarks often focus on narrow domains or simplified tasks that lack the diversity, realism, and long-horizon complexity required to evaluate agents' real-world performance. To address this gap, we introduce the Tool Decathlon (dubbed as Toolathlon), a benchmark for language agents offering diverse Apps and tools, realistic environment setup, and reliable execution-based evaluation. Toolathlon spans 32 software applications and 604 tools, ranging from everyday...
