[2509.16952] AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation

Computer Science > Computation and Language arXiv:2509.16952 (cs) [Submitted on 21 Sep 2025 (v1), last revised 30 Mar 2026 (this version, v2)] Title:AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation Authors:Tiancheng Huang, Ruisheng Cao, Yuxin Zhang, Zhangyi Kang, Zijian Wang, Chenrun Wang, Yijie Luo, Hang Zheng, Lirong Qian, Lu Chen, Kai Yu View a PDF of the paper titled AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation, by Tiancheng Huang and 10 other authors View PDF HTML (experimental) Abstract:The growing volume of academic papers has made it increasingly difficult for researchers to efficiently extract key information. While large language models (LLMs) based agents are capable of automating question answering (QA) workflows for scientific papers, there still lacks a comprehensive and realistic benchmark to evaluate their capabilities. Moreover, training an interactive agent for this specific task is hindered by the shortage of high-quality interaction trajectories. In this work, we propose AirQA, a human-annotated comprehensive paper QA dataset in the field of artificial intelligence (AI), with 13,956 papers and 1,246 questions, that encompasses multi-task, multi-modal and instance-level evaluation. Furthermore, we propose ExTrActor, an automated framework for instruction data synthesis. With three LLM-based agents, ExTrActor can perform example generation and trajectory collection without human intervent...