[2510.00507] Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphs

Computer Science > Computation and Language arXiv:2510.00507 (cs) [Submitted on 1 Oct 2025 (v1), last revised 5 Mar 2026 (this version, v3)] Title:Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphs Authors:Yurun Chen, Xavier Hu, Yuhan Liu, Ziqi Wang, Zeyi Liao, Lin Chen, Feng Wei, Yuxi Qian, Bo Zheng, Keting Yin, Shengyu Zhang View a PDF of the paper titled Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphs, by Yurun Chen and 10 other authors View PDF HTML (experimental) Abstract:As multimodal LLM-driven agents advance in autonomy and generalization, traditional static datasets face inherent scalability limitations and are insufficient for fully assessing their capabilities in increasingly complex and diverse tasks. Existing studies have attempted to generate agent tasks using LLMs, but due to the inherent hallucinations of LLMs and the lack of internal data relationship modeling, these tasks often exhibit semantic inconsistencies and solvability issues. To address these challenges, we introduce Graph2Eval, a knowledge-graph-driven framework for automated, scalable, and semantically grounded agent task generation. At its core, Graph2Eval leverages a knowledge graph built from heterogeneous external data sources as a structured task space, generating multimodal agent tasks through subgraph sampling and task construction guided by task templates and meta-path strategies. To further ensure task reliability, a multi-s...