[2603.27333] ComBench: A Repo-level Real-world Benchmark for Compilation Error Repair

Computer Science > Software Engineering arXiv:2603.27333 (cs) [Submitted on 28 Mar 2026] Title:ComBench: A Repo-level Real-world Benchmark for Compilation Error Repair Authors:Jia Li, Zeyang Zhuang, Zhuangbin Chen, Yuxin Su, Wei Meng, Michael R. Lyu View a PDF of the paper titled ComBench: A Repo-level Real-world Benchmark for Compilation Error Repair, by Jia Li and 5 other authors View PDF HTML (experimental) Abstract:Compilation errors pose pervasive and critical challenges in software development, significantly hindering productivity. Therefore, Automated Compilation Error Repair (ACER) techniques are proposed to mitigate these issues. Despite recent advancements in ACER, its real-world performance remains poorly evaluated. This can be largely attributed to the limitations of existing benchmarks, \ie decontextualized single-file data, lack of authentic source diversity, and biased local task modeling that ignores crucial repository-level complexities. To bridge this critical gap, we propose ComBench, the first repository-level, reproducible real-world benchmark for C/C++ compilation error repair. ComBench is constructed through a novel, automated framework that systematically mines real-world failures from the GitHub CI histories of large-scale open-source projects. Our framework contributes techniques for the high-precision identification of ground-truth repair patches from complex version histories and a high-fidelity mechanism for reproducing the original, ephemeral ...