![[2602.22638] MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.22638] MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios
Summary
MobilityBench introduces a benchmark for evaluating LLM-based route-planning agents, addressing challenges in real-world mobility scenarios through a systematic evaluation framework.
Why It Matters
This research is significant as it provides a structured approach to assess the performance of route-planning agents, which are increasingly important in enhancing human mobility. By addressing existing evaluation gaps, MobilityBench aims to improve the reliability and effectiveness of AI-driven navigation tools, ultimately benefiting users in diverse urban environments.
Key Takeaways
- MobilityBench offers a scalable benchmark for evaluating LLM-based route-planning agents.
- The benchmark is built from real user queries, ensuring relevance to actual mobility scenarios.
- A deterministic API-replay sandbox is designed to enhance reproducibility in evaluations.
- Current models excel in basic tasks but struggle with preference-constrained route planning.
- The research includes publicly available benchmark data and evaluation tools.
Computer Science > Artificial Intelligence arXiv:2602.22638 (cs) [Submitted on 26 Feb 2026] Title:MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios Authors:Zhiheng Song, Jingshuai Zhang, Chuan Qin, Chao Wang, Chao Chen, Longfei Xu, Kaikui Liu, Xiangxiang Chu, Hengshu Zhu View a PDF of the paper titled MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios, by Zhiheng Song and 8 other authors View PDF HTML (experimental) Abstract:Route-planning agents powered by large language models (LLMs) have emerged as a promising paradigm for supporting everyday human mobility through natural language interaction and tool-mediated decision making. However, systematic evaluation in real-world mobility settings is hindered by diverse routing demands, non-deterministic mapping services, and limited reproducibility. In this study, we introduce MobilityBench, a scalable benchmark for evaluating LLM-based route-planning agents in real-world mobility scenarios. MobilityBench is constructed from large-scale, anonymized real user queries collected from Amap and covers a broad spectrum of route-planning intents across multiple cities worldwide. To enable reproducible, end-to-end evaluation, we design a deterministic API-replay sandbox that eliminates environmental variance from live services. We further propose a multi-dimensional evaluation protocol centered on outcome validity, complemented by assessments ...
