![[2508.01780] LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2508.01780] LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?
Summary
The paper presents LiveMCPBench, a benchmark designed to evaluate the capabilities of agents using Model Context Protocol (MCP) tools in real-world scenarios, highlighting performance gaps and challenges in tool retrieval and composition.
Why It Matters
As AI models increasingly rely on external tools, understanding their effectiveness in diverse, real-world tasks is crucial. LiveMCPBench addresses the limitations of current evaluations by providing a comprehensive framework that assesses agent performance across multiple tools and servers, paving the way for improvements in AI tool integration.
Key Takeaways
- LiveMCPBench evaluates 95 real-world tasks to assess MCP tool usage.
- There is a significant performance gap among state-of-the-art LLMs, with success rates varying from 30% to 78.95%.
- Retrieval errors are a major bottleneck, accounting for nearly half of task failures.
- The benchmark includes a tool suite of 70 servers with 527 tools for reproducibility.
- Active tool composition is strongly correlated with task success.
Computer Science > Artificial Intelligence arXiv:2508.01780 (cs) [Submitted on 3 Aug 2025 (v1), last revised 26 Feb 2026 (this version, v2)] Title:LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools? Authors:Guozhao Mo, Wenliang Zhong, Jiawei Chen, Qianhao Yuan, Xuanang Chen, Yaojie Lu, Hongyu Lin, Ben He, Xianpei Han, Le Sun View a PDF of the paper titled LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?, by Guozhao Mo and 9 other authors View PDF HTML (experimental) Abstract:Model Context Protocol (MCP) has become a key infrastructure for connecting LLMs with external tools, scaling to 10,000+ MCP servers with diverse tools. Unfortunately, there is still a large gap between real-world MCP usage and current evaluation: they typically assume single-server settings and directly inject tools into the model's context, bypassing the challenges of large-scale retrieval and multi-tool composition. To bridge this gap, we propose LiveMCPBench, which evaluates 95 real-world daily tasks explicitly constructed to stress diverse tools and scaled multi-server routing. The benchmark includes a ready-to-deploy tool suite of 70 servers with 527 tools, ensuring reproducibility without scattered API configuration. We further introduce an LLM-as-a-Judge evaluation framework that directly verifies task outcomes, handling dynamic data sources and multiple valid solution paths. We benchmark 12 state-of-the-art LLMs and observe a substantial performance gap: while Claude-Sonnet-4 reac...
