[2603.01562] RubricBench: Aligning Model-Generated Rubrics with Human Standards

Computer Science > Artificial Intelligence arXiv:2603.01562 (cs) [Submitted on 2 Mar 2026] Title:RubricBench: Aligning Model-Generated Rubrics with Human Standards Authors:Qiyuan Zhang, Junyi Zhou, Yufei Wang, Fuyuan Lyu, Yidong Ming, Can Xu, Qingfeng Sun, Kai Zheng, Peng Kang, Xue Liu, Chen Ma View a PDF of the paper titled RubricBench: Aligning Model-Generated Rubrics with Human Standards, by Qiyuan Zhang and 10 other authors View PDF HTML (experimental) Abstract:As Large Language Model (LLM) alignment evolves from simple completions to complex, highly sophisticated generation, Reward Models are increasingly shifting toward rubric-guided evaluation to mitigate surface-level biases. However, the community lacks a unified benchmark to assess this evaluation paradigm, as existing benchmarks lack both the discriminative complexity and the ground-truth rubric annotations required for rigorous analysis. To bridge this gap, we introduce RubricBench, a curated benchmark with 1,147 pairwise comparisons specifically designed to assess the reliability of rubric-based evaluation. Our construction employs a multi-dimensional filtration pipeline to target hard samples featuring nuanced input complexity and misleading surface bias, augmenting each with expert-annotated, atomic rubrics derived strictly from instructions. Comprehensive experiments reveal a substantial capability gap between human-annotated and model-generated rubrics, indicating that even state-of-the-art models struggle...