[2603.00993] CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration

Computer Science > Artificial Intelligence arXiv:2603.00993 (cs) [Submitted on 1 Mar 2026] Title:CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration Authors:Yiyue Qian, Shinan Zhang, Yun Zhou, Haibo Ding, Diego Socolinsky, Yi Zhang View a PDF of the paper titled CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration, by Yiyue Qian and 5 other authors View PDF HTML (experimental) Abstract:Large Language Models (LLMs) have revolutionized AI-generated content evaluation, with the LLM-as-a-Judge paradigm becoming increasingly popular. However, current single-LLM evaluation approaches face significant challenges, including inconsistent judgments and inherent biases from pre-training data. To address these limitations, we propose CollabEval, a novel multi-agent evaluation framework that implements a three-phase Collaborative Evaluation process: initial evaluation, multi-round discussion, and final judgment. Unlike existing approaches that rely on competitive debate or single-model evaluation, CollabEval emphasizes collaboration among multiple agents with strategic consensus checking for efficiency. Our extensive experiments demonstrate that CollabEval consistently outperforms single-LLM approaches across multiple dimensions while maintaining robust performance even when individual models struggle. The framework provides comprehensive support for various evaluation criteria while ensuring efficiency through its collaborative design. Subjects: Artificial...